Shell Script to monitor folder and upload found files via FTP


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell Script to monitor folder and upload found files via FTP
# 1  
Old 07-24-2010
Shell Script to monitor folder and upload found files via FTP

Hi everyone!

I'm in a need of a shell script that search for all files in a folder, move all those files to a temp folder, and upload those files via FTP. When the file transfer via FTP completes successfully, the file is moved to a completed folder. In case any of those files fails, the file will be moved to a failed folder.

I have very little knowledge on Shell Scripting, but I'll try to explain as clear as possible what I need using a example flow below.

Code:
#!/bin/sh

# Configure variables at the begining of the script
MONITOR_DIR=/path/to/monitor/files/
TMP_DIR=/path/to/monitor/files/temp/
SUCCESS_DIR=/path/to/monitor/files/success/
FAILED_DIR=/path/to/monitor/files/failed/
PROCCESS_ID_DIR={execution timestamp string}
FILE_EXT=flv

# FTP variables
FTP_HOST=ftp.myserver.com
FTP_USER=mysername
FTP_PASS=mypassword
FTP_REMOTE_DIR=/


The script will work with these steps:
  1. For each file with extension $FILE_EXT in $MONITOR_DIR
  2. echo "found file: " and the name(s) of file(s) found
  3. Move all found files to $TMP_DIR/$PROCCESS_ID_DIR (folder $PROCCESS_ID_DIR has to be created)
  4. For each file in $TMP_DIR/$PROCCESS_ID_DIR
  5. Upload via FTP each file individually using the FTP variables
  6. If FTP transfer completes, move that file to $SUCCESS_DIR, else move it to $FAILED_DIR
  7. echo "file (file_name) transfered" or "file (file_name) failed" depending on what happened above
  8. When all the files in $TMP_DIR/$PROCCESS_ID_DIR are processed, delete $PROCCESS_ID_DIR folder in $TMP_DIR
  9. quit script

That's basically it. I need this to monitor/backup/copy-to-CDN files uploaded to my site using a cron file.

Any help will be greatly appreciated...

Thanks!

Smilie

---------- Post updated 07-24-10 at 12:19 AM ---------- Previous update was 07-23-10 at 10:04 AM ----------

Created this script

Code:
#!/bin/sh

# Configure variables at the begining of the script
MONITOR_DIR=/var/www/web12/web/php-site-monitor/files
TMP_DIR=/var/www/web12/web/php-site-monitor/files/tmp
SUCCESS_DIR=/var/www/web12/web/php-site-monitor/files/success
FAILED_DIR=/var/www/web12/web/php-site-monitor/files/failed
FILE_EXT=flv

# FTP variables
FTP_HOST=upload-ftp.simplecdn.com
FTP_USER=myuser
FTP_PASS=mypass
FTP_REMOTE_DIR=/

cd $MONITOR_DIR
COUNT_FILES=$(ls -l *.$FILE_EXT | grep ^- | wc -l)

if [ $COUNT_FILES -gt 0 ]; then

echo "A total of" $COUNT_FILES "file(s) found.";
echo "Creating TEMP directory:" $TMP_DIR/$PROCCESS_ID_DIR;

mkdir $TMP_DIR/$PROCCESS_ID_DIR

for f in *.$FILE_EXT; do
echo "Moving file:" $f "to:" $TMP_DIR/$PROCCESS_ID_DIR;
mv $f $TMP_DIR/$PROCCESS_ID_DIR
done

cd $TMP_DIR/$PROCCESS_ID_DIR

for f in *.$FILE_EXT; do
echo "Uploading file via FTP:" $f
ftp -in $FTP_HOST <<EOF
user $FTP_USER $FTP_PASS
binary
cd $FTP_REMOTE_DIR
put $f
bye
EOF

if [ $? -eq 0 ]
then
	echo "FTP File Upload Completed:" $f
	mv -f $f $SUCCESS_DIR
else
	echo "FTP File Upload Failed:" $f
	mv -f $f $FAILED_DIRs
fi

done

echo "Uploads completed...";
echo "Deleting TEMP folder:" $TMP_DIR/$PROCCESS_ID_DIR;

rm -rf $TMP_DIR/$PROCCESS_ID_DIR

else

echo "No files found... Exit!";

fi

It does the work...

Any recommendations?

Last edited by pulsorock; 07-23-2010 at 11:20 AM..
# 2  
Old 07-24-2010
or...
yum -y install inotify-tools

while true; do inotifywait -r -e MODIFY dir/ && <script>; done;

or

while inotifywait -r -e MODIFY dir/; do make; done;
# 3  
Old 07-24-2010
The script is to be run as a cron in Rackspace Cloud Sites, so that's the only option I have.
# 4  
Old 07-24-2010
Sorry I couldn't be of more help.
Some one with more Kung-Fu will be along to assist.

Have a Great Day!
# 5  
Old 07-24-2010
You to... Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Request for Shell script to move files from Subfolder to Parent folder and delete sub folder

Hi Team, I am new to shell script and there is a requirement where files should be moved from Subfolder to parent folder. Eg: parent folder --> /Interface/data/test/IN Sub folder -->/Interface/data/test/IN/Invoice20180607233338 Subfolder will be always with timestamp... (6 Replies)
Discussion started by: srivarun15
6 Replies

2. Homework & Coursework Questions

Need shell script to monitor files

Hi Experts, I am not good in writing script. Just stared. I am looking for shell script to check following parameters. 1) Number of files on remote Linux SUSE server.- Any directory and sub directory. 2) I should define number of files in script. Files should be variable. 3) Age of the... (2 Replies)
Discussion started by: ApmPerfMonitor
2 Replies

3. Shell Programming and Scripting

Shell script for FTP folder pattern

Hello, I have 3 folders on FTP server in the format yyyymmddhhmiss as below,ftp> cd /home/walgreens 250 CWD command successful ftp> ls 200 PORT command successful 150 Opening ASCII mode data connection for file list 20150708072901 20150708092901 20150708102901The above folders are in... (1 Reply)
Discussion started by: pavan_test
1 Replies

4. Shell Programming and Scripting

FTP in shell script and selecting files for upload

Hi, Im a newbie with programming and shell scripting. Im running OSX/Mac and Darwin. I would like to create a shell script that would : 1. Search a Volume and directory (including subdirectories) for a file that : * filename ends with ”_Highres.pdf” and * the file creation date of... (8 Replies)
Discussion started by: NickeZ28
8 Replies

5. Shell Programming and Scripting

Shell Script for Upload/download files using cURL

hi please help me out here, i want to use curl command in shell script to test web pages, what i have is an opening page, when i click on a button on opening page, the next page comes up and then i have to upload a file n then click another button to submit and then comes the output page,... (2 Replies)
Discussion started by: Olivia
2 Replies

6. Shell Programming and Scripting

FTP from Unix Shell script to Windows Shared folder ?

Hi Before Posting my query in this forum, I have gone through various similar postings to get some idea on ftp and how to do that from unix shell script to windows server. My question is related to FTP'ing from Unix to windows shared folder My basic question is 1. Is it possible to do FTP... (4 Replies)
Discussion started by: shekharjchandra
4 Replies

7. Shell Programming and Scripting

ftp put in shell script -- whole file doesn't upload

Hi I'm having some trouble with a bash shell script that I'm writing. In the script, I'm trying to upload a file to a backup repository using ftp, but the whole file doesn't get uploaded. This is the file's properties at the start (I've highlighted the file size in red): -rw-r--r-- 1 root... (2 Replies)
Discussion started by: Viola
2 Replies

8. Shell Programming and Scripting

Shell script to monitor tmp folder for uploads

Hello, We have been having some issues with our users overwriting files, and then not having a backup. What I would love to do, is create a shell script to monitor /tmp, for uploads, and make a copy of the file they are trying to upload before the upload finishes. Is this possible at all? (6 Replies)
Discussion started by: mrfr0g
6 Replies

9. UNIX for Dummies Questions & Answers

ftp error 550 folder not found

Hi, I am having an ftp script(runs in HP unix server) that connect to another server where we have different folders named as email addresses(eg."anarayana@yahoo.com"). Then it goes to each folder and put the file in that folder. I found an error when remote folder is not created. "550... (1 Reply)
Discussion started by: lnarayana
1 Replies

10. UNIX for Advanced & Expert Users

Ftp Upload Folder

Hi! I'd like to upload a complete folder with subfolders with put. At the moment I always get the err-msg: aboutme: not a plain file Could anyone help me?? Thanx (3 Replies)
Discussion started by: roberthawke
3 Replies
Login or Register to Ask a Question