Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

howto monitor a directory for files then sftp them

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting howto monitor a directory for files then sftp them
# 1  
Old 03-09-2011
howto monitor a directory for files then sftp them
Morning all
I hope I have put this in the correct forum.

I have a requirement to monitor a directory on a server for files being sftp'ed in and then to sftp them of to another server. The issues I have though of are making sure the files have completely transferred onto the server before they are picked up and sent on.

Can anyone help with how to do this as my scripting abilities are limited? If you need me to explain more just say

Thanks

Lee
# 2  
Old 03-09-2011
Here are 2 possible solutions :

Either you copy the file in a TMP folder on the remote system and you move them in the final destination folder after the transfer via a remote command.
or you transfer all your files and at the end you sftp a flag file (e.g. tsf.ok). The script on the remote server should check the flag exists before processing the files.
# 3  
Old 03-09-2011
If you have control over how files are put onto the remote server then implementing a rename policy is the best way:
Code:
put file.txt file.new
rename file.new file.txt

If not you will have to monitor the directory to see if the new files are changing. I would suggest getting lftp as it is a scriptable client that can be used with different protocols and has a mirror command.

You would keep a local list of files in that directory and download a new list periodically, comparing the two lists. When there are new files your script then checks more frequently until the size of the new files stops changing. Then the copying could be done.

I could give you a copy of an lftpls script I've written to give you a start

Andrew
# 4  
Old 03-09-2011
apmcd47
If you could send a copy may be a good start for me.

To fill in some blanks
In a nut shell our server is acting like a relay. We will relieve a file/s from one company and send them straight back out onto a second company. I have asked why use us but thats how its to be. So I dont know when the files are coming in or number of.

I am lead to believe they are going to be small txt files possibly pgp'ed.

I will have to do this twice and it will have to go back the other way as well.

Lee
# 5  
Old 03-09-2011
Could this help you? Below script will check the size of the file. Once it found file is not growing then do SFTP with -b option. Create a file ftp_command.ftp
with ftp commands like below.
Code:
#cat ftp_command.ftp
cd /tmp
put filename
bye

Script
Code:
#!/bin/sh
size=$(stat --format=%s filename)
sleep 5
cur_size=0
while (($cur_size != $size))
do
cur_size=$(stat --format=%s filename)
sleep 10
done

sftp -b /my/dir/ftp_command.ftp user@server

# 6  
Old 03-09-2011
I wrote this as a ksh script, but I think it will work as bash instead.
Code:
#!/bin/ksh
# usage: lftpls -u user -d directory system

user=

usage() {
   echo "Usage: ${0##*/} [-u user] [-d directory] system"
}

while getopts u:d:w:h opt
do
   case $opt in
      u) user="-u $OPTARG"   ;;
      d) rdir="cd $OPTARG ;" ;;
      h) usage; exit         ;;
   esac
done
shift `expr $OPTIND - 1`

if (($# != 1))
then
   echo I need a remote system
   usage
   exit
fi

PREFIX_CMD="set cmd:fail-exit yes;set ftp:passive-mode false;"  
        # (1) Causes lftp to exit on failure
        # (2) force the PORT rather than PASV command

lftp ${user} -e "${PREFIX_CMD} ${rdir} nlist ; quit" $1

Note that nlist is equivalent to ls -1 - you may need to replace with cls to change the behaviour of this command, which I called lftpls (a name I stole from the ncftp suite, which contains an ncftpls command).

Example for sftp:
Code:
lftpls -u fred sftp://bedrock.com/home/fred/flintstone

Lists the contents of /home/fred/flintstone as user fred on sftp server bedrock.com. This version requires the ssh keys to be set up.

Andrew
# 7  
Old 03-10-2011
Thanks al for the replies, I will have a play around with the info you have given. the requirements have changed today.
Here is a brief explanation:
file sent to me which is PGP'ed. I then have to send it on to third party. They will send back a file to me which I them pass back to the original sender, They check it and send a second file which is either for a success or failure.
Im having a think as to the best way to do this. The simple one of send it between the 2 missing me is not an option.
The 4 types of files have unique names eg
prd_numbers_xxxx_xxxx_xxx
prd_reply_xxxx_xxxx_xxx
prd_success_xxxx_xxxx_xxx
prd_failure_xxxx_xxxx_xxx

I know this is more than my original post but if anyone has any pointers I would appreciate it

Many thanks

Lee
Image
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

Chroot sftp users, remote sftp login shows wrong timestamp on files

Hello, I have a weird issue, I have RHEL 5.7 running with openssh5.2 where sftpgroup OS group is chroot. I see the difference difference in timestamp on files, when I login via ssh and SFTP, I see four hour difference, is something missing in my configuration. #pwd... (8 Replies)
Discussion started by: bobby320
8 Replies

2. UNIX for Dummies Questions & Answers

Q: Howto compare 2 files

Greetings, I made an extraction on 2 different databases. What I need to do is to compare those extractions to know what is on database1 which is not on database2 and vice versa. In those files, there are only numbers. So each line is just a number witch should be present on both file. If... (6 Replies)
Discussion started by: Sekullos
6 Replies

3. AIX

AIX ftp/sftp script monitor to failed logins

Hi All, Any idea on how to write a script on AIX 5.3 to monitor ftp or sftp login failed. Thanks and more power, Itik (2 Replies)
Discussion started by: itik
2 Replies

4. Shell Programming and Scripting

Directory Monitor in KSh

Hi, I need to write a directory monitor, i.e. a korn shell script which would Report changes to the directory contents, like: added file1, deleted file2, updated file3 , created subdir (optional)... There is no specific file pattern. So far I have written a little script... (1 Reply)
Discussion started by: olegkon
1 Replies

5. Solaris

Restricting SFTP user to a defined directory and home directory

Hi, I've created solaris user which has both FTP and SFTP Access. Using the "ftpaccess" configuration file options "guest-root" and "restricted-uid", i can restrict the user to a specific directory. But I'm unable to restrict the user when the user is logged in using SFTP. The aim is to... (1 Reply)
Discussion started by: sftpuser
1 Replies

6. AIX

monitor directory events

I'm am looking for a cheap way to trigger a script when a new file is written in a specific directory. AIX 5.3. It is a production system, so no kernel patching (i.e. inotify). Filemon and audtiing are too expensive. Thanks in advance. (2 Replies)
Discussion started by: pbillast
2 Replies

7. Shell Programming and Scripting

Monitor capacity of directory

Good morning. I have been attempting to find a way to monitor the capacity of a directory so that when it reaches 80% or higher I can send an event. I was able to find a script that does this for the whole drive by I can not seem to figure out how to do this for just a single directory. ... (1 Reply)
Discussion started by: LRoberts
1 Replies

8. Shell Programming and Scripting

script to monitor files in a directory and sending the alert

Hi All, We are having important config files in an directory which was accessable by all /auto/config/Testbed/>ls config1.intial config2.intial config3.inital often we find that some of the lines are missing in config files, we doubt if some one is removing. I would like to write... (0 Replies)
Discussion started by: shellscripter
0 Replies

9. Shell Programming and Scripting

script to monitor directory

What is the best way for a script to run to monitor a directory for the presence of files and then perform a function afterwords? I was hoping to have it continually run and sleep until it detects that files are present in the directory, then break out of the loop and go on to the next step. ... (17 Replies)
Discussion started by: nulinux
17 Replies

10. UNIX for Dummies Questions & Answers

Howto install RPM package for non-Root user directory?

Dear expert, Suppose I have an application that comes in rpm format. Let's call it: my_downloaded_package.rpm How can I install it in my particular home directory (since I'm not a root user). Say the directory where I want to install above rpm package is: ~/.my_desired_location... (1 Reply)
Discussion started by: monkfan
1 Replies
Login or Register to Ask a Question