Automated FTP task


 
Thread Tools Search this Thread
Special Forums IP Networking Automated FTP task
# 22  
Old 02-06-2006
"Iam using this part of script in a shell script...
The Tee wont work bcos Iam not just doing ftp."
Well you lost me there... Smilie

"I want the output to be on screen and to a logfile."
I understand that. It's gonna be a bit rough...
Code:
#! /usr/bin/ksh

#
# get password

print -n password -
stty -echo
read PASSWD
stty echo
echo

HOST=someftphost
USER=joeuser

#
# arrange for all stdout and stderr to be sent to a tee process
exec 5>&1
tee /dev/tty > ftptest.log |&
pid1=$!
exec  >&p 2>&1 4>&1

#
# invoke ftp co-process
ftp -nv >&4 2>&4 |&
pid2=$!
print -p open $HOST
print -p user $USER $PASSWD
print -p get foo
print -p bye
wait $pid2

#
# disconnect stdout and stderr from tee process
# send a line to stdout which comes directly from this script
exec >&- 2>&- 4>&- >&5 2>&1
echo
exit 0

# 23  
Old 03-16-2006
Hello Perderabo,

I'd like to ask for further assitance in this regards, If you dont mind.

I have a similar requirement where in I have :-

a) Predefined the desitnation server / hardcoded value.
b) Ask for the TAR file to transfer to the destination server.
c) After the file has been transfered it has to be extracted.

How can i do this and the important question is, how will the extraction and other validations take place via FTP ?

Kindly assist.

Thanks
# 24  
Old 10-30-2006
what about security

Perderabo Thanksa lot for the script example , in the script
HOST=damien1
USER=con
PASSWD=
is there a way to mask this password . The reason being that i do the transfer as myself which in this case I have to expose my password, is there a way to encrypt this.
Thanks for your help guys.
# 25  
Old 11-01-2006
These examples are great and sort of do what I want.
Since I'm not a great scriptor can one of you please help me out.

This is what I have.

- System A is a Solaris server
- System B is a Windows server
- Both systems have an outbound and an inbound ftp dir
- I need to move the lastest file from SystemA outbound to System B inbound dir and also from System B outbound to System A inbound
- The script will be run out of cron on System A
- Both these servers will have files in both the outbound and inbound dir. The script needs to only transfer any new files

I have been told I must use ftp and not rsync as rsync is not allowed through the firewalls but ftp is.
Tornado
# 26  
Old 11-07-2006
Its OK..
I've worked out how to do it...
Tornado
# 27  
Old 11-30-2006
Using the info gathered from this thread, this is what I have so far..
Code:
#! /usr/bin/ksh
#
# Goran Cvetanoski - 07/11/2006
#
# /export/ftpdir/autoftp
#
# This script uses ftp to move files from one server to another.
# It copies files from one servers Outgoing to another servers Incoming
# It FTPs to Incoming/.receiving directory and then moves the files to Incoming
# Once the files have been sent it moves the files from Outgoing to Outgoing/.sent
# It deletes files older then 14 days from both servers Outgoing/.sent directories
# It traps any FTP errors and send an email with the error message
# It can also send emails on successfull transfers, if wanted
#
# Available flags:
# -------------------------------------------------------------------------
# mget - ftp mget from $RHOST:\$ROUT to $LIN
# mput - ftp mput from $LOUT to $RHOST:\$RIN
# xfer - Perform a mget then an mput on $RHOST
#
# Two log files are created:
# -------------------------------------------------------------------------
# ftp.mail - This gets emailed to $ADMIN , if comment removed
# ftp.log  - Verbose output from the ftp
#
# -------------------------------------------------------------------------
#                  A small note on all the fd stuff..
# -------------------------------------------------------------------------
# exec 4>&1
# ftp -nv >&4 2>&1 |&
#
# fd 0 is standard-in, fd 1 is standard-out, and fd 2 is standard-error.
# The line "exec 4&>1" opens fd 4 and assigns it to whatever fd 1 was
# assigned to. I am sorta "saving a copy of fd 1 in fd 4".
#
# The line "ftp -nv >&4 2>&1 |&" is a little harder.
#
# The "|&" turns the process into a co-process that allows subsequent
# "print -p" statements to send lines to the co-process' standard-in and
# "read -p" to read from its standard-out. So ksh forks a copy of itself
# and fiddles with the fd's 0 and 1 until this it set-up. But it leaves
# the rest of the fd's alone.
#
# Then it encounters ">&4" which causes it to set the ftp process' standard
# out to whatever 4 is. Well since 4 is a copy of 1 before the co-process,
# we are back to writing to the original shell's standard out. Lastly,
# the 2>&1 does the same thing for standard error.
#
# CHANGE LOG
# =========================================================================
# 07/11/2006 - Goran Base script created
# 09/11/2006 - Goran Created usage function
# 24/11/2006 - Goran Added error checking of ftp transfers
# 27/11/2006 - Goran Added ftptest function
# 01/12/2006 - Goran Added cleanup function
# 25/10/2007 - Goran Added CheckDir function
#

LHOST=`uname -n`
RHOST=123.456.789.123
USER=Enter your userid here
PASSWD=Enter your password here

ADMIN=unix-admin@mydomain.com

TSTFILE=TEST.TXT

LOG=/var/adm/ftp.log
LOG1=/tmp/ftp.mail
LOG2=/tmp/ftp.log
LOG3=/tmp/ftp.parselog

LIN=/export/ftpdir/Incoming
LOUT=/export/ftpdir/Outgoing
RIN=/$USER/Incoming
ROUT=/$USER/Outgoing


usage ()
{
    echo ""
    echo ""
    echo "      #########################################################"
    echo "      ##                                                     ##"
    echo "      ######    Usage: autoftp [ mget | mput | xfer ]    ######"
    echo "      ##                                                     ##"
    echo "      #########################################################"
    echo "      ##                                                     ##"
    echo "      ##  mget: Copy files from $RHOST to $LHOST  ##"
    echo "      ##                                                     ##"
    echo "      ##  mput: Copy files from $LHOST to $RHOST  ##"
    echo "      ##                                                     ##"
    echo "      ##  xfer: Perform an mget and an mput from $LHOST    ##"
    echo "      ##        to $RHOST                           ##"
    echo "      ##                                                     ##"
    echo "      #########################################################"
    echo ""
    echo ""
}

SendMail()
{
    SUBJECT=$1
    OUTPUT=$2

    cat $OUTPUT | mailx -s "$SUBJECT" $ADMIN
}

CheckDir()
{
    DIR=$1

    if [ ! -d $DIR ]; then
        echo "Error - Directory $DIR does not exist" >> $LOG2
        echo "ERROR: Directory Error - $DIR does not exist" | mailx -s "Directory Error - $DIR" $ADMIN
        exit 1
    fi
}

doftp ()
{
    CMD=$1
    LOCAL=$2
    REMOTE=$3

if [[ $CMD = "mget" ]]; then
    echo "================================================" >> $LOG2
    echo "==    Receiving Files From $RHOST    ==" >> $LOG2
    echo "================================================" >> $LOG2
else
    echo "============================================" >> $LOG2
    echo "==    Sending Files to $RHOST    ==" >> $LOG2
    echo "============================================" >> $LOG2
fi
    exec 4>&1
    ftp -nv >&4 2>&1 |&

    pid2=$!
    print -p open $RHOST
    print -p user $USER $PASSWD
    print -p binary
    print -p lcd $LOCAL
    print -p cd $REMOTE
    print -p prompt
    print -p $CMD \*
    print -p bye
    wait $pid2
}

ftptest ()
{
    mv $LIN/$TSTFILE $LOUT

    echo "=======================================" >> $LOG2
    echo "== $TSTFILE being sent back to $RHOST ==" >> $LOG2
    echo "=======================================" >> $LOG2

    exec 4>&1
    ftp -nv >&4 2>&1 |&

    pid4=$!
    print -p open $RHOST
    print -p user $USER $PASSWD
    print -p binary
    print -p lcd $LOUT
    print -p cd $RIN/.receiving
    print -p put $TSTFILE
    print -p bye
    wait $pid4
}

log ()
{
    CMD=$1
    SRC=$2

    echo "=== $CMD Files Report ===" >> $LOG1
    cd $SRC
    echo "Size(KB)  File" >> $LOG1
    ls -s | grep -v total | awk '{ $size = $1 / 2; print $size "    "  $2 }' | sort -n >> $LOG1
    echo "=============================" >> $LOG1
}

mvfiles ()
{
    LOCAL=$1
    LDEST=$2
    REMOTE=$3
    RDEST=$4

    exec 4>&1
    ftp -n >&4 2>&1 |&

    pid3=$!
    print -p open $RHOST
    print -p user $USER $PASSWD
    print -p binary
    print -p cd $REMOTE
    cd $LOCAL
    for files in `ls` ; do
       print -p rename $files $RDEST/$files
       mv $files $LDEST
    done
    print -p bye
    wait $pid3
}

chkerror ()
{
    cat /dev/null > $LOG3
    cat $LOG2 >> $LOG
    grep -v bytes $LOG2 >> $LOG3

    for error in `awk '{print $1}' $LOG3`; do
        case $error in
            Not) SendMail "FTP Error - between $LHOST and $RHOST" $LOG2
                 exit 1;;
            332) SendMail "FTP Error - Need account for login" $LOG2
                 exit 1;;
            350) SendMail "FTP Error - Requested file action pending further information" $LOG2
                 exit 1;;
            421) SendMail "FTP Error - Service not available, closing control connection" $LOG2
                 #This may be a reply to any command if the service knows it
                 #must shut down.
                 exit 1;;
            425) SendMail "FTP Error - Can't open data connection" $LOG2
                 exit 1;;
            426) SendMail "FTP Error - Connection closed; transfer aborted" $LOG2
                 exit 1;;
            450) SendMail "FTP Error - Requested file action not taken" $LOG2
                 #File unavailable (e.g., file busy).
                 exit 1;;
            451) SendMail "FTP Error - Requested action aborted: local error in processing" $LOG2
                 exit 1;;
            452) SendMail "FTP Error - Requested action not taken" $LOG2
                 #Insufficient storage space in system.
                 exit 1;;
            500) SendMail "FTP Error - Syntax error, command unrecognized" $LOG2
                 #This may include errors such as command line too long.
                 exit 1;;
            501) SendMail "FTP Error - Syntax error in parameters or arguments" $LOG2
                 exit 1;;
            502) SendMail "FTP Error - Command not implemented" $LOG2
                 exit 1;;
            503) SendMail "FTP Error - Bad sequence of commands" $LOG2
                 exit 1;;
            504) SendMail "FTP Error - Command not implemented for that parameter" $LOG2
                 exit 1;;
            530) SendMail "FTP Error - Not logged in" $LOG2
                 exit 1;;
            532) SendMail "FTP Error - Need account for storing files" $LOG2
                 exit 1;;
            550) SendMail "FTP Error - No such file or directory" $LOG2
                 #File unavailable (e.g., directory empty, no access).
                 exit 1;;
            551) SendMail "FTP Error - Requested action aborted: page type unknown" $LOG2
                 exit 1;;
            552) SendMail "FTP Error - Requested file action aborted" $LOG2
                 #Exceeded storage allocation (for current directory or
                 #dataset).
                 exit 1;;
            553) SendMail "FTP Error - Requested action not taken -  File name not allowed" $LOG2
                 exit 1;;
        esac
    done
}

cleanup ()
{
    echo "=======================================" >> $LOG1
    echo "=======================================" >> $LOG2
    echo "==    Cleaning up old sent files     ==" >> $LOG1
    echo "==    Cleaning up old sent files     ==" >> $LOG2
    echo "=======================================" >> $LOG1
    echo "=======================================" >> $LOG2
    echo " Files removed from $LHOST:$LOUT/.sent" >> $LOG1
    echo " Files removed from $LHOST:$LOUT/.sent" >> $LOG2

    ## Clean up local servers Outgoing folder
    cd $LOUT/.sent
    find $LOUT/.sent -mtime +14 >> $LOG1
    find $LOUT/.sent -mtime +14 >> $LOG2
    find $LOUT/.sent -mtime +14 -exec rm {} \;

    ## Create the .history folder
    cat /dev/null > $LIN/.history/.list
    cd $LIN
    for history in `ls` ; do
        touch $LIN/.history/$history
    done

    exec 4>&1
    ftp -n >&4 2>&1 |&

    pid5=$!
    print -p open $RHOST
    print -p user $USER $PASSWD
    print -p binary
    print -p cd $ROUT/.sent
    cd $LIN/.history
    find $LIN/.history -mtime +14 >> $LIN/.history/.list
    for old in `cat $LIN/.history/.list` ; do
        print -p delete $old    # Delete file from remote server
        rm $old                 # Delete file from local folder
    done
    print -p bye
    wait $pid5

    echo " Files removed from $RHOST:$ROUT/.sent" >> $LOG1
    echo " Files removed from $RHOST:$ROUT/.sent" >> $LOG2
    cat $LIN/.history/.list >> $LOG1
    cat $LIN/.history/.list >> $LOG2
    echo "" >> $LOG2
}

## Create log headers here
echo "=============================" > $LOG1
echo "=============================" > $LOG2
echo " `date`" >> $LOG1
echo " `date`" >> $LOG2
echo "=============================" >> $LOG1
echo "=============================" >> $LOG2

CheckDir $BASE
CheckDir $LIN
CheckDir $LOUT

## Create required local directories
test ! -d $LIN/.receiving && mkdir $LIN/.receiving
test ! -d $LIN/.history && mkdir $LIN/.history
test ! -d $LOUT/.sent && mkdir $LOUT/.sent

CheckDir $LIN/.receiving
CheckDir $LIN/.history
CheckDir $LOUT/.sent

exec 5>&1

# Switch the comments on the next 2 lines if
# you also want to see output on the screen 
#tee /dev/tty >> $LOG2 |&
tee >> $LOG2 |&

pid1=$!
exec  >&p 2>&1 4>&1

case $1 in

mget)   doftp mget $LIN/.receiving $ROUT
        log Recieved $LIN/.receiving
        mvfiles $LIN/.receiving .. $ROUT .sent ;;

mput)   doftp mput $LOUT $RIN/.receiving
        log Sent $LOUT
        mvfiles $LOUT .sent $RIN/.receiving .. ;;

xfer)   doftp mget $LIN/.receiving $ROUT
        log Recieved $LIN/.receiving
        mvfiles $LIN/.receiving .. $ROUT .sent
        doftp mput $LOUT $RIN/.receiving
        log Sent $LOUT
        mvfiles $LOUT .sent $RIN/.receiving .. ;;

   *)   usage
        exec >&- 2>&- 4>&- >&5 2>&1
        exit 1 ;;
esac

## Run ftptest if test file exists in remote Outgoing folder
if [[ -e "$LIN/$TSTFILE" ]]; then
    ftptest $tstfile
    mvfiles $LOUT .sent $RIN/.receiving ..
    exec >&- 2>&- 4>&- >&5 2>&1
    chkerror
    SendMail "FTP Test Results - between $LHOST and $RHOST" $LOG2
    exit 0
fi

exec >&- 2>&- 4>&- >&5 2>&1

## Remove sent files older then 14 days
## From local and remote servers
cleanup
chkerror

#Remove comment below to receive an email on successfull ftp
#SendMail "FTP Results - between $LHOST and $RHOST" $LOG1

exit 0

This is writen for a Solaris 8 system, thats why I have to do the ls -s and awk stuff and can't just use ls -h in my log function. I also need to remove all files older then 14 days from the .sent directory. This is easy on the server as I just use find, but its a bit more tricky on the remote Windows NT system.

UPDATED
I have updated the code above to include my cleanup function..

Last edited by Tornado; 10-08-2007 at 07:21 PM..
Tornado
# 28  
Old 11-30-2006
I haven't read the entire thread, but when it comes to automated FTP transfers ( every day to backup machine ) use rsync.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Automated FTP

Hello,I just know the basics of scripting & I need a script to automate ftp since I use it quite often. I use ftp to transfer different kind of files and everytime the source & the destination directories are different for transferring files, so can anyone help me out on this urgently.. (2 Replies)
Discussion started by: mohit_02mar
2 Replies

2. Shell Programming and Scripting

Automated FTP script using .netrc to multiple FTP servers

Hi all, I'm using the following script to automated ftp files to 1 ftp servers host=192.168.0.1 /usr/bin/ftp -vi >> $bkplog 2>&1 <<ftp open $host bin cd ${directory} put $files quit ftp and the .netrc file contain machine 192.168.0.1 login abc... (4 Replies)
Discussion started by: varu0612
4 Replies

3. Shell Programming and Scripting

Automated FTP

I want to do Automated FTP from linux client to LINUX server using a shell script after every one hour. And copies last most updated file from Linux server. OR You can say that whenever files get modified on LINUX server so it also copies on my LINUX client. Linux SERVER path=... (4 Replies)
Discussion started by: wakhan
4 Replies

4. Shell Programming and Scripting

how to devlop automated FTP in UNIX

Automated FTP. hint : use 'atd' to schedule to run a specific script. An Env Variable should be created,say CONF_DIR which points to some dir. @ some time, create a TAR file of this and FTP it to some server. (3 Replies)
Discussion started by: vishalzone2002
3 Replies

5. Shell Programming and Scripting

automated ftp.

Hi I am trying to delete some specific files ( files other than created today) from the server on a cron basis. I wrote a small script, but I am stuck up in how to delete only specific files. #!/usr/bin/expect -f set IP set timeout -1 spawn ftp $IP expect ): send "username\n"... (10 Replies)
Discussion started by: sangfroid
10 Replies

6. IP Networking

Automated ftp for Multiple files

I have seen the script posted yesterday for automated ftp Can we do some thing like ftp ing multiple files in one script Example input.txt has all files names to be ftped input.txt ------ a.tar b.ccp c.perl i need to ftp all the files present in input.txt i tried something like... (0 Replies)
Discussion started by: pbsrinivas
0 Replies

7. UNIX for Dummies Questions & Answers

one time automated ftp session

How can an automated script be setup to run at a later time in the day. cron runs recurring tasks. I am interested in a one time process. I want to back up a number of files whenever I make changes to an application and ftp for backup purposes. The script to automate the zipping of files has... (1 Reply)
Discussion started by: msmkeml
1 Replies

8. UNIX for Advanced & Expert Users

Automated FTP

My requiremnet is to write a FTP script which will ftp a file to a specified ftp url. How can I automate the process usnig the unix cron. (11 Replies)
Discussion started by: shauche
11 Replies

9. UNIX for Dummies Questions & Answers

FTP automated?

If I wanted a machine to put a specific file onto another OS far across the internet via FTP - and I wanted to do it automatically not user intervented, how would I do that? Use the PUT command? The file name and position never changes, it gets overwritten and the system on the other end... (6 Replies)
Discussion started by: n9ninchd
6 Replies
Login or Register to Ask a Question