Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Newbie needs help with some bash scripting

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Newbie needs help with some bash scripting
# 1  
Old 02-04-2017
Newbie needs help with some bash scripting
Please bear with me, I'm a beginner but have had some experience and decent knowledge to understand things as I read them and I'm in the process of trying to learn more. I recently inherited a UNIX server which has a bash script which is part of a crontab schedule that needs to be modified (or potentially replaced at this point)


Here's what I understand
Essentially, the bash script takes the data that is being sent to the server and after it is successfully received, runs a Unix2Dos command against it -- which doesn't seem to work due to an older version, as well are removes line fees in certain files. It then takes the data, renames it to have a unix2dos extension, sends it out via FTP to another server. It then archives the original data, and the data that has the Unix2Dos command run against it, by running a gzip command against it and then moves it to a folder with a YYYYMM format (example /folder1/Data/VENDOR/inbound/archive/format/201702)

Here is what they now need
Since the Unix2Dos process is not working, we have opted to use another server for the process but the data is still coming through this server and will always be coming into here. We just now don't want to FTP over to another one, or run the Unix2Dos command or remove any line feeds. However, we want to do the following now through a crontab job:

Takes the original data that is being sent to the server, without renaming it, and after it is successfully received, copy it over to a folder (/folder1/Data/VENDOR/inbound/CompanyArchive/format/Unix2Dos) for an outside FTP job to pick up. Only specific file names can be placed here. It then needs to gzip the data, move it over to a folder with a YYYYMM format (example /folder1/Data/VENDOR/inbound/CompanyArchive/format/201702)





I know it seems simple enough, but I have no idea how to do this based on my very limited knowledge of Unix, bash scripting and anything else. I'll share the script and hope many can try to help me here. Thanks in advance for any help provided

Here is our crontab schedule
Code:
# crontab /var/spool/cron/crontabs/username
# crontab -l
#ident  "@(#)username       1.00    05/08/09"
#
5 2,5,8,11,14,16,17,20 * * * bash /folder1/Data/bin/SplitMoveFiles.sh 2>/dev/null
1 11,21 * * * bash /folder1/Data/bin/MoveFormatFiles.sh 2>/dev/null
#1 * * * * bash /folder1/Data/bin/RenameFileAsSoonAsItIsDownloaded.sh 2>/dev/null

This is the bash script I'm focusing on MoveFormatFiles.sh
(sorry, had to modify this a bit to remove company names and such.. hopefully someone can help me with what I am trying to accomplish)



Code:
#!/usr/local/bin/bash
# Created by User on 20170204
#
# we are now receiving files in a  format for client data from vendor
# These files need to have their line feed replaced by carriage return + line feed
# The files will need to be posted to a subfolder in the SFTP site
 
# get date in the form MCDYMMDD
FormattedDate="`date -u +%Y%m%d`"
YearMonth="`date -u +%Y%m`"
# get date formatted with seconds
# log file path
MasterLog="/folder1/Data/FOLDER2/Place_Log/Client_"${FormattedDate}".log"
 
pid=$$
FilesPostedToClient="/d01/Data/ListOfPostedFiles."${pid}
CompanyFTPConnect="/d01/Data/CompanyFTPConnect."${pid}
 
# initiate with commands to connect to FTP site
echo "cd CLIENT_FORMAT" > ${CompanyFTPConnect}
echo "lcd /folder/Data/Vendor/inbound" >>  ${CompanyFTPConnect}
 
##########################################################
#Function that adds all available files to FTP command
##########################################################
funCheckIfThereAreFilesToSend(){
    for sFilesToSend in `cd /folder1/Data/"${1}"/inbound; ls -d "${2}"*.u2d`
    do
         echo "put ${sFilesToSend}" >>  ${CompanyFTPConnect}
    done
}
 
#############################################
# Function that creates a folder if it does not exist
#############################################
funCreateNewFolder(){
    sFolderPath="/folder1/Data/VENDOR/inbound/CompanyArchive/Format/"${YearMonth};
 
    if [ ! -e  ${sFolderPath} ];
    then
        mkdir ${sFolderPath};
    fi;
}
##############################################
#function to look for new files received from VENDOR
# make a copy of original file and store it as a compressed file.
##############################################
funHandleNewFormatFiles(){
    for sNewFile in `cd /folder1/Data/"${1}"/inbound; ls -d "${2}"*|/opt/sfw/bin/gawk '{c=split($0,a,".");if(a[c]!="u2d"){print $0}}'`
    do
        #Create folder that will house the archived informtaion if it does not exist
        funCreateNewFolder
 
        cd /folder1/Data/"${1}"/inbound
        echo "File " ${sNewFile} " was received from VENDOR." >> ${MasterLog}
 
        # compress and store/backup
        /usr/bin/gzip -q -c /folder1/Data/"${1}"/inbound/${sNewFile} >> \
            /folder1/Data/VENDOR/inbound/CompanyArchive/Format/${YearMonth}/${sNewFile}"_"${FormattedDate}.gz
        #Create a quick copy for now (comment the line below after a couple of months without issues)
        cp /folder1/Data/"${1}"/inbound/${sNewFile}  /folder1/Data/"${1}"/inbound/temp/
 
        # convert to DOS format
        #unix2dos /folder1/Data/"${1}"/inbound/${sNewFile}  /folder1/Data/"${1}"/inbound/${sNewFile}.u2d
        #/bin/awk '{printf "%s\r\n", $0}' /folder1/Data/"${1}"/inbound/${sNewFile} > /folder1/Data/"${1}"/inbound/${sNewFile}.u2d
        /opt/sfw/bin/gawk 'BEGIN{ORS="\015\012"}{print $0}' /folder1/Data/"${1}"/inbound/${sNewFile} > /folder1/Data/"${1}"/inbound/${sNewFile}.u2d
        #/bin/sed -e 's/$/\r/' /folder1/Data/"${1}"/inbound/${sNewFile} > /folder1/Data/"${1}"/inbound/${sNewFile}.u2d
        #tr -d "\000" < /folder1/Data/"${1}"/inbound/${sNewFile} > /folder1/Data/"${1}"/inbound/${sNewFile}.tr
        #/bin/perl -pe 's|\n|\r\n|' /folder1/Data/"${1}"/inbound/${sNewFile}.tr > /folder1/Data/"${1}"/inbound/${sNewFile}.u2d
 
        # if for some reason unix2dos does not see the need to convert and leaves as it found it, we'll go ahead and rename
        if [ ! -e /folder1/Data/"${1}"/inbound/${sNewFile}.u2d ];
        then
             mv -f /folder1/Data/"${1}"/inbound/${sNewFile} /folder1/Data/"${1}"/inbound/${sNewFile}.u2d
        fi;
 
        # get rid of orig
        rm -f /folder1/Data/"${1}"/inbound/${sNewFile}  2>/dev/null
 
        # we now want to make sure we remove any duplicate carriage returns (0d) - seen them in the past
        if [ -e /folder1/Data/"${1}"/inbound/${sNewFile}.u2d ];
        then
             tr -s "\015" < /folder1/Data/"${1}"/inbound/${sNewFile}.u2d > /folder1/Data/"${1}"/inbound/${sNewFile}.tr
             mv -f /folder1/Data/"${1}"/inbound/${sNewFile}.tr /folder/Data/"${1}"/inbound/${sNewFile}.u2d
        fi;
 
        # call function that adds any available u2d files
        funCheckIfThereAreFilesToSend $1 $2
    done
}
###########################################################
#Call above function for all dataset names and vendor paths
#FunctionName - VendorFolderName - DataSetName
##########################################################
funHandleNewFormatFiles VENDOR 006620.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 050720.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 111840.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 160800.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 185510.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 200120.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 221310.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 277310.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 277910.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 284370.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 323530.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 338600.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 347130.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 598200.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 730100.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 803220.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 809520.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 821720.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 850410.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 940210.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR A48275.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR AV8887.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR E03989.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR F90855.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR LU1178.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR LY5444.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR NP7727.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR PCBS.S.ONE.C1
funHandleNewFormatFiles VENDOR PCBS.V.ONE.C1
funHandleNewFormatFiles VENDOR QA0052.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR RE1939.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR RX1961.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR RZ7223.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SE3158.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SK9344.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SK9481.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SL0261.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SL1683.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SL3757.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SL5383.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SL9121.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SM2403.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SM2448.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SN6707.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SN8793.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SP7431.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SR2668.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR ST3975.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR ST6484.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR PCBS.I.ONE.C100106.C0
funHandleNewFormatFiles VENDOR PCBS.O.ONE.C100106.C0
funHandleNewFormatFiles VENDOR 000000.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 003070.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 030300.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 142650.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 214500.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 362200.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 452140.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 622900.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 754550.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR 754550.VENDOR.VTNS.I7
funHandleNewFormatFiles VENDOR 788700.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR 840530.VENDOR.VTNS.C0
funHandleNewFormatFiles VENDOR DX2038.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR GH3849.VENDOR.SDN.C0
funHandleNewFormatFiles VENDOR RQ2307.VENDOR.SDN.C0
funHandleNewFormatFiles VENDOR SP1967.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SY3025.VENDOR.DCS.C6
funHandleNewFormatFiles VENDOR SY3151.VENDOR.DCS.C6
 
 
##########################################################
# connect to the FTP site and verify that the files have been successfully posted; only then, archive them.
##########################################################
 
#production ftp site
#echo "password!" > ${CompanyFTPConnect}
 
echo "ls "  >> ${CompanyFTPConnect}
echo "quit" >> ${CompanyFTPConnect}
 
###################################
#Connect to Production server, post files and get list of files posted
##################################
/usr/bin/sftp username@ftp.site.com < ${CompanyFTPConnect} > ${FilesPostedToClient}
 
#cat ${CompanyFTPConnect}                #DEBUG
#cat "${FilesPostedToClient}"           #DEBUG
 
##########################################################
# archive all files that may have been posted
##########################################################
for sUploadedFile in `cat "${FilesPostedToCompany}"|/opt/sfw/bin/gawk '{cLineSplit=split($0,aLineSplit);\
    if((aLineSplit[cLineSplit] ~ ".VENDOR.DCS.C6")||(aLineSplit[cLineSplit] ~ ".VENDOR.VTNS.C0")||(aLineSplit[cLineSplit] ~ "PCBS.S.ONE.C1")||(aLineSplit[cLineSplit] ~ "PCBS.V.ONE.C1")||(aLineSplit[cLineSplit] ~ ".I.ONE.C100106.C0")||(aLineSplit[cLineSplit] ~ ".O.ONE.C100106.C0")||(aLineSplit[cLineSplit] ~ "VENDOR.VTNS.I7")||(aLineSplit[cLineSplit] ~ "VENDOR.SDN.C0")){print aLineSplit[cLineSplit]}}'`
do
    echo "File " ${sUploadedFile} " was successfully posted to Company sFTP site \
and will now get moved to the /archive folder" >> ${MasterLog}
 
    if [ -r /folder1/Data/VENDOR/inbound/${sUploadedFile} ]; then
 
        /usr/bin/gzip -q -c /folder1/Data/*/inbound/${sUploadedFile}  > \
            /folder1/Data/VENDOR/inbound/CompanyArchive/Format/${YearMonth}/${sUploadedFile}.gz
        rm -f /folder1/Data/*/inbound/${sUploadedFile}
    fi
done
 
rm "${CompanyFTPConnect}" 2>/dev/null
rm "${FilesPostedToClient}" 2>/dev/null

Last edited by Scrutinizer; 02-04-2017 at 01:27 PM.. Reason: code tags
# 2  
Old 02-04-2017
To help you we need to know a bit more... like your environment:
What architecture, what OS and version etc...
now:
Code:
1 11,21 * * * bash /folder1/Data/bin/MoveFormatFiles.sh 2>/dev/null

Why call bash as it is mentioned as shell to be used in your script ?
Unix2Dos is standard but not on all UNIX and may not have that name ( unix2dos, ux2dos...) without more clues about your OS hmmm....
# 3  
Old 02-05-2017
I can't see a reason why unix2dos should not work, even an older version. Does it if you run it interactively? Does it work on your data files, or mayhap on highly simplified test files?
For clarity, you might want to simplify your - overcomplicated - script as well, and add some decent error checking and logging.
# 4  
Old 02-05-2017
Quote:
Originally Posted by vbe
To help you we need to know a bit more... like your environment:
What architecture, what OS and version etc...
now:
Code:
1 11,21 * * * bash /folder1/Data/bin/MoveFormatFiles.sh 2>/dev/null

Why call bash as it is mentioned as shell to be used in your script ?
Unix2Dos is standard but not on all UNIX and may not have that name ( unix2dos, ux2dos...) without more clues about your OS hmmm....
Quote:
Originally Posted by RudiC
I can't see a reason why unix2dos should not work, even an older version. Does it if you run it interactively? Does it work on your data files, or mayhap on highly simplified test files?
For clarity, you might want to simplify your - overcomplicated - script as well, and add some decent error checking and logging.
Thank you both for responding. As far as I can see, it's Sun Microsystems on SunOS 5.9 running Solaris 2.6

As I mentioned, I inherited this but not necessarily inherited the server. I inherited the functions and processes it does. To give an idea of it's main function, it's responsible for receiving a lot of data company wide through a NDM connection. It's one of those servers that needs to be up almost all the time.

From what I know, the user that is running this script is not able to run it as shell but can run it as bash, if that makes any sense at all (sorry if it doesn't, I am fairly new to understand this)

With that said, I should be more clear. It's not that the unix2dos command doesn't work. It does work. It just doesn't work in every case for how they want it to work. The person who wrote this code, made it overly complicated because the person who requested this type of data was also making things complicated. In some cases, line feeds were being left in there after the unix2dos command was ran, but they needed to be removed. In other cases, the line feeds needed to stay in there in certain spots so trying to figure out what was and wasn't needed was a challenge.

Myself and one other coworker figured out that the unix2dos command on this server was older, and wasn't doing everything it was supposed to but we weren't able to get a newer version on there. However, he had the latest on a server he had which worked 100% the way they wanted it to and essentially now he just wants to pick up the data from our server and run the command from there (we can't move the NDM process over, unfortunately, as much as I'd love to... mainly because it does other company functions that actually DO work on this one)

Hope this makes sense and thank you both for the responses.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash scripting - Newbie

Hi all, I have drill to do and I'll very appreciate your help: Please create a simple CSV file as follow (3 columns and 4 rows): A,B,C A,”B,C”,D “A,B”,C,D o A,B,”C,D” - Please refer to the comma between the quotation marks as a parameter and not as a separator. - Please provide... (3 Replies)
Discussion started by: elior
3 Replies

2. Shell Programming and Scripting

Shell Scripting Newbie

Hi Guys, I want to create a shell script to run multiple jobs in sequence. Explaination - If I were to run each jobs individually I would have gone to folder - "abin"(where my shellscript is place) as follows cd abin abin > runappeng.sh abc001 Now, I have list of programs which are like... (8 Replies)
Discussion started by: chaits84
8 Replies

3. Shell Programming and Scripting

Newbie bash scripting help requested

Hi, I'm very new to bash scripting and Linux in general. I'm running Ubuntu Server 10.04 and trying to write a bash script to launch a program. In doing so, I've come across a couple of things that I obviously don't understand. Here is a short script that exemplifies those things: ... (9 Replies)
Discussion started by: Carson Dyle
9 Replies

4. Shell Programming and Scripting

Scripting needed for newbie

Hi, I am newbie in shell scripting I have a file name like simple.txt which comes from Mainframe systems onto windows dir every 15 minutes daily. File name is same. Every 15 minutes it updates. I need to write shell script to check if the file arrived every 15 min or not. If the new file... (4 Replies)
Discussion started by: chinniforu2003
4 Replies

5. Shell Programming and Scripting

Shell Scripting NEWBIE - Need Help

Could someone please recommend a very good shell scripting book for me. I would be starting a new job that would require a very good understanding of shell scripting. Please help. (3 Replies)
Discussion started by: ayoka
3 Replies

6. Shell Programming and Scripting

scripting newbie... some help please?

hi all, i am just getting in to bash scripting, so don't be too harsh... i've created this little backup script, and it's just awfull... ugly, doesn't work like I want it to, the works. anyways, i was hoping some of you might help me improve it and learn a little in the process. what i... (13 Replies)
Discussion started by: jmd9qs
13 Replies

7. Shell Programming and Scripting

scripting newbie needs help

I have written a script that will email a generic user when a device is "offline". I would like to enhance this by having the script lookup a contact's email and automatically add it to the MAIL_LIST. I am trying to lookup and return data based on a field common in two files File 1 ... (0 Replies)
Discussion started by: irishluck66
0 Replies

8. Shell Programming and Scripting

Scripting Newbie

Seems simple but I am having difficulty with this one: I am trying to write a single command line argument (which will be a path) - the program should print out the owner of the path. I can not get anything I write to run. Please help. (5 Replies)
Discussion started by: Kymmers7
5 Replies

9. UNIX for Dummies Questions & Answers

Shell Scripting Newbie

I'm relatively new at this scripting game, just need to learn some basic stuff for data handling. My current need is to write a script that loops through a textfile of filenames, and for each file removes the first line and re-writes that file to a new name. In fact I could do with knowing... (1 Reply)
Discussion started by: mattyjim2
1 Replies

10. Shell Programming and Scripting

Newbie - Need help in bash scripting

Hi there, I am a student and currently working on a project. I have a file that contains about 50 filenames. (1.txt, 2.txt, 3.txt ...). I would like to know how can I store these filenames into a variable using a loop? I would appreciate if anyone can help me. Thank You. Regards, Bib (4 Replies)
Discussion started by: coolbib
4 Replies
Login or Register to Ask a Question