Will shell script executes in sequence


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Will shell script executes in sequence
# 1  
Old 02-09-2015
Will shell script executes in sequence

I have a shell script scheduled in cron job to run at every 1 minute which transfers files to remote machine and then move the files to backup folder.

Code:
cd /u01/app/ftp_tmp
sftp user@hostname <<-EOF
cd /home/user/ftp
mput *
bye
EOF
mv /u01/app/ftp_tmp/* /u01/app/ftp_bkp

Now the problem is it transfers the file to remote machine incompletely, but it has the completed file in backup folder.
So I wanted to know how it comes to the last command before completing the file transfer. Please assist.

Last edited by rbatte1; 02-09-2015 at 11:42 AM.. Reason: Added CODE tags and corrected spelling and case
# 2  
Old 02-09-2015
Hi,
please use code tags for your future posts.
Remove the dash before EOF.

hth

Last edited by Don Cragun; 02-11-2015 at 05:27 AM.. Reason: Remove duplicate post.
# 3  
Old 02-09-2015
Several issues here:-
  • When will files be arriving? You might not have a complete file when the send starts.
  • You have no error checking, so in the event of failure you code will continue and move the file.
  • If you have hundreds of files or some large ones and the next transfer starts (you schedule every minute) what will happen with the second set of transfers when the first one completes and moves the files?
I hope that this gives you something to consider in your process.



Robin
# 4  
Old 02-10-2015
Now I kept an exit code status after the file transfer , and it returns the exit code as 0
But still the file which is transferred is incomplete

Code:
cd /u01/app/ftp_tmp
sftp user@hostname <<EOF
cd /home/user/ftp
mput *
bye
EOF
if [ $? -eq 0 ]
then
echo "File transferred successfully"
else
echo "Partial File transfer : $?"
fi
mv /u01/app/ftp_tmp/* /u01/app/ftp_bkp

Moderator's Comments:
Mod Comment
Please use CODE tags for code, files, input & output/errors.
It makes it far easier to read and preserves multiple spaces in case these are important.

Last edited by rbatte1; 02-10-2015 at 08:53 AM.. Reason: Added CODE tags
# 5  
Old 02-10-2015
You are getting the return code of the whole SFTP. Because of the timing risks, your code may see what you consider a partial file and successfully transfer that before the file is complete - and you get a zero return code.

You would be better ensuring that a flag-file to confirm the file is complete is created and then setting up a loop to do the following for each flag-file:-
  • Delete the flag file
  • Send one file
  • Move that file to the archive
If there is a lot to transfer which may take more than a minute and your code is scheduled every minute, you may also need some sort of locking mechanism to ensure that your script is not running two versions at the same time.



Robin
# 6  
Old 02-11-2015
I noticed that files larger than this are getting transferred, the issue is only with one type of file, so not sure whether the file got corrupted or not.
But each time the file gets generated with the timestamp, so the file name changes each and every time with only the third qualifer as same.

I checked with md5sum for the file both source & destination.
But since the file size in source & destination are different how to identify whether the file is corrupted or not.
# 7  
Old 02-11-2015
As Robin has already said, it looks like you may have two or more copies of your cron script running simultaneously working on the same file. And, you are almost certainly starting to transfer some files before they are ready to be copied (giving you a partial copy of your file on the remote system).

You need to fix your code so that a file will only be transferred once and only after the entire file is in place and is ready to be transferred.

We don't have nearly enough information to tell you exactly how to do that since you haven't given us any details about how the files are prepared, how they are written, copied, or linked into the directory from which they are transferred; nor how you are trying to keep two or more copies of your cron job from working on the same file simultaneously.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Print Line as per the dependent sequence in shell script.

Hi i have a file like this as shown below: DA PROCESS_ID IDENTIFIER DA_FILE STATUS WAITING_FOR SCOPED_DEPENDENT 1836 21000 01052019 BH90P.TEMP.DA1836.FTP W NULL ... (6 Replies)
Discussion started by: krishnaswarnkar
6 Replies

2. UNIX for Dummies Questions & Answers

Script partially executes??

Hi All, I am calling a shell script from another shell script, however, it only executes part of it (the echo commands only). What could be some causes for that? For example: ShellScriptA.sh: ... ... ... . ShellScriptB.sh ShellScriptB.sh contents: echo date echo... (7 Replies)
Discussion started by: DBnixUser
7 Replies

3. UNIX for Advanced & Expert Users

Checking missing data's sequence (shell script | UNIX command)

Dear All members, i have some trouble here, i want to ask your help. The case is: I have some data, it's like: -ABCD1234 -ABCD1235 -ABCD1237 -BCDE1111 -BCDE1112 -BCDE1114 there is some missing data's sequence (the format is: ABCD = name 1234 = sequence). I want to print the... (2 Replies)
Discussion started by: septian.tri
2 Replies

4. Shell Programming and Scripting

Shell script not getting called through cron job but executes fine manually.

Hi, My shell script not getting called through cron job. The same works fine when executed manually. I tried to generate logs to find if the scripts has some errors related to path using following command- trying to execute .sh file every 5 mins: */5 * * * * /home/myfolder/abc.sh... (17 Replies)
Discussion started by: Dejavu20
17 Replies

5. Shell Programming and Scripting

script executes some time but not always.

I have following script to send email when a USB is attached. #!/bin/bash NUMBER=`/bin/cat /u/numberoflines` LINES=`/usr/bin/wc -l < /var/log/messages` DIFFERENCE=$(($LINES-$NUMBER)) if ; then tail -n $DIFFERENCE /var/log/messages |while read line do if $( echo $line | grep --quiet... (2 Replies)
Discussion started by: kashif.live
2 Replies

6. UNIX for Advanced & Expert Users

Sequence number merged with hypen, shell script

Hello Folks, i have to write shell scripting for given expected output manner. in given input we have to write shell script in such a way that sequence no can b merged/link between start and end digit with hyphen "-" symbol and rest of digit separated by "," Eg : For Input "2 6 7 8 11 12... (9 Replies)
Discussion started by: panchalh
9 Replies

7. Windows & DOS: Issues & Discussions

Batch script executes twice

Hi, Batch script gets executed without any error, but on execution some of the partial contents of the batch file gets appended at the end of the file which is currently in execution, hence the script tries to execute again for the second time , which should not happen. How to get it... (5 Replies)
Discussion started by: milink
5 Replies

8. Shell Programming and Scripting

root executes a script as another user

Hi All, Am using the below command to start my application using the root user su - bin -c "/home/bin/test/start.sh" but am getting the error becaue i have set some environment varibales in bin's .profile when i execute the command start.sh by logging directly into bin account it's... (9 Replies)
Discussion started by: ravi.sri24
9 Replies

9. Shell Programming and Scripting

shell script executes program, but waits for a prompt

Sorry, newbie here. I have the following shell script which basically executes the sh-n-body.i686 program a specified number of times. However, before the sh-n-body.i686 begins its calculations it prompts for input from the user. In this case the user would have press ". return" and... (7 Replies)
Discussion started by: lionatucla
7 Replies

10. Shell Programming and Scripting

crontab and shell script that executes a sql.

Problem: I have a crontab and when it kicks off, xxx.sh shell script is called. Which has a nohup sqlplus session call. Problem is sql does not get executed and a text file is not getting created. Only a empty log file is getting created. Are there any constraints for crontab to open a sql... (6 Replies)
Discussion started by: radhika
6 Replies
Login or Register to Ask a Question

Featured Tech Videos