FTP Cron Issue - Shell Programming and Scripting

edua · #1 11-26-2007

I have an automated FTP script that is giving me grief. The script runs a ftp sweep for one of our accounts. It gets about 90 files then hangs. When i run ftp manually from cli everything runs great and I pickup the thousands of files waiting.

I used to run lftp, but it has been behaving even worse. Here is a snipet from my shell script that runs the automated pickup.

Code:

# OLD WAY!
#/usr/local/bin/lftp -d -f $SESSION_CMDS

# this way works from CLI, but stops short when in this script
ftp -f -d -i -n $SERVER <<END_SCRIPT
quote USER $USERNAME
quote PASS $PASSWORD
mget *
quit
END_SCRIPT

I've run this on both FreeBSD 6.2 and Gentoo 2007.0. Both with the same results. Termination and loop back to trying to login again after 90 files or so, but then I get the User already logged in message.

I thought maybe this was firewall related, but since I can run ftp from bash and have no problems, I'm lost.

Please if anyone knows any special voodoo, I would greatly appreciate it

Does FTP have a file limit (count or size) running from cron? If anyone can help and you need more info please let me know. THANKS!

porter · #2 11-26-2007

Have you thought of using $HOME/.netrc for the username and password?

edua · #3 11-26-2007

good pointer. I should have used .netrc file for my cron job. But I don't think this will help with my problem. Its almost like there is a file limit on count or size when running via cron. but i'm not sure thats true either, just what I'm observing.

porter · #4 11-26-2007

1. are you setting the current directory?

2. can you get any output from ulimit?

Code:

     limit, ulimit, unlimit - set or get limitations on the  sys-
     tem resources available to the current shell and its descen-
     dents

edua · #5 11-26-2007

1. Yes I'm setting the current dir ($WORKING_DIR)

Code:

# change the working directory and print it
cd $WORKING_DIR
echo "Chaning to FTP download directory: "
pwd
echo ""

# Start output for ftp session
echo "FTP Session:"

# Call LFTP and pass in the commands as a parameter
#/usr/local/bin/lftp -d -f $SESSION_CMDS
#lftp -d -f $SESSION_CMDS
ftp -d -i -n $SERVER <<END_SCRIPT
quote USER $USERNAME
quote PASS $PASSWORD
mget *
quit
END_SCRIPT

# Put in newlines for next entry
echo " "
echo " "

# Terminate the program
exit 0

2. ulimit = unlimited

Lakris · #6 11-26-2007

Maybe this doesn't answer Your question, but have You considered trying to use wget or curl or some other powerful "getter"?
eg:
wget ftp://${USERNAME}:${PASSWORD}@${SERVER}/*

Cameron · #7 11-26-2007

More suggestion rather than solution ...
The USERNAME & PASSWORD should be hidden in an environment file and run at the start of the script. Just a thought.

ftp -inv ${CFS_DNS} <${FTP_BATCH} >${FTP_LOG}

Would humbly suggest:
a) Create an actual batchjob ($FTP_BATCH)
b) drop the `quote user` for `user`

Don't know what OS your using, `-d` isn't used on HP-UX 11.23.

Cheers,
Cameron

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Cron issue	kingdbag	SUN Solaris	5	07-10-2008 08:14 AM
Another Cron issue.	Beefy	UNIX for Dummies Questions & Answers	2	03-13-2008 10:30 AM
Issue with cron job -Please help	bsandeep_80	Shell Programming and Scripting	6	07-27-2007 11:20 AM
Issue with cron job	nattynatty	Shell Programming and Scripting	1	11-21-2005 04:37 PM
cron issue	Optimus_P	UNIX for Advanced & Expert Users	5	08-26-2002 10:30 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode