Visit Our UNIX and Linux User Community


FTP Cron Issue


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting FTP Cron Issue
# 1  
Old 11-26-2007
FTP Cron Issue

I have an automated FTP script that is giving me grief. The script runs a ftp sweep for one of our accounts. It gets about 90 files then hangs. When i run ftp manually from cli everything runs great and I pickup the thousands of files waiting.

I used to run lftp, but it has been behaving even worse. Here is a snipet from my shell script that runs the automated pickup.

Code:
# OLD WAY!
#/usr/local/bin/lftp -d -f $SESSION_CMDS

# this way works from CLI, but stops short when in this script
ftp -f -d -i -n $SERVER <<END_SCRIPT
quote USER $USERNAME
quote PASS $PASSWORD
mget *
quit
END_SCRIPT

I've run this on both FreeBSD 6.2 and Gentoo 2007.0. Both with the same results. Termination and loop back to trying to login again after 90 files or so, but then I get the User already logged in message.

I thought maybe this was firewall related, but since I can run ftp from bash and have no problems, I'm lost.

Please if anyone knows any special voodoo, I would greatly appreciate it Smilie Does FTP have a file limit (count or size) running from cron? If anyone can help and you need more info please let me know. THANKS!
# 2  
Old 11-26-2007
Have you thought of using $HOME/.netrc for the username and password?
# 3  
Old 11-26-2007
good pointer. I should have used .netrc file for my cron job. But I don't think this will help with my problem. Its almost like there is a file limit on count or size when running via cron. but i'm not sure thats true either, just what I'm observing.
# 4  
Old 11-26-2007
1. are you setting the current directory?

2. can you get any output from ulimit?

Code:
     limit, ulimit, unlimit - set or get limitations on the  sys-
     tem resources available to the current shell and its descen-
     dents

# 5  
Old 11-26-2007
1. Yes I'm setting the current dir ($WORKING_DIR)

Code:
# change the working directory and print it
cd $WORKING_DIR
echo "Chaning to FTP download directory: "
pwd
echo ""

# Start output for ftp session
echo "FTP Session:"

# Call LFTP and pass in the commands as a parameter
#/usr/local/bin/lftp -d -f $SESSION_CMDS
#lftp -d -f $SESSION_CMDS
ftp -d -i -n $SERVER <<END_SCRIPT
quote USER $USERNAME
quote PASS $PASSWORD
mget *
quit
END_SCRIPT

# Put in newlines for next entry
echo " "
echo " "

# Terminate the program
exit 0

2. ulimit = unlimited
# 6  
Old 11-27-2007
Maybe this doesn't answer Your question, but have You considered trying to use wget or curl or some other powerful "getter"?
eg:
wget ftp://${USERNAME}:${PASSWORD}@${SERVER}/*
# 7  
Old 11-27-2007
More suggestion rather than solution ...
The USERNAME & PASSWORD should be hidden in an environment file and run at the start of the script. Just a thought.

ftp -inv ${CFS_DNS} <${FTP_BATCH} >${FTP_LOG}

Would humbly suggest:
a) Create an actual batchjob ($FTP_BATCH)
b) drop the `quote user` for `user`

Don't know what OS your using, `-d` isn't used on HP-UX 11.23.

Cheers,
Cameron

Previous Thread | Next Thread
Test Your Knowledge in Computers #36
Difficulty: Easy
Moving processes from the main memory of a memory-constrained Linux computer to disk is called 'swapping'.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Cron issue

i have written a script to sftp yesterday's logs from another server as below: cd /export/home/abc/xyz/tt d=`gdate -d'yesterday' +%Y%m%d` sftp abc@XXX.XX.XX.XX<<EOF cd /yyy/logs/archive mget abc.log.$d* EOF cd /export/home/abc/xyz/scripts nohup ./ss.sh PROD & it is working fine... (2 Replies)
Discussion started by: ssk250
2 Replies

2. Solaris

CRON Issue

Hello, I am running Solaris 8. I have set a cron job that runs every couple hours. If I run the script manually, it runs just fine (logged in as root). The cron however will not run. It is producing an rc=1 error. Any thoughts would be appreciated. Thanks (4 Replies)
Discussion started by: jkmtm
4 Replies

3. Red Hat

cron issue

Hello, Having and issue with a job scheduled in cron. The script: #!/bin/bash 2 3 # Example shell script which can be added to roots cron job to check the 4 # Embedded Satellite disk space usage. If any table is over 90% usage, send 5 # a notice to the default email address... (2 Replies)
Discussion started by: mgb
2 Replies

4. Solaris

Cron issue

When I list whats in cron -l its fine but when I try to -e edit it...it returns a number 309 can't you not edit cron this way with solaris 10? I can do it fine in sol 8 and 9. export EDITOR="vi" is set in my profile I am using BASH $ sudo crontab -l Password: #ident "@(#)root ... (5 Replies)
Discussion started by: kingdbag
5 Replies

5. UNIX for Dummies Questions & Answers

Another Cron issue.

Hey guys, I have a script called my_test: # !/usr/bin/sh `touch /usr/test/me` The script has been saved in /usr/test/my_test I have executed chmod 755 /usr/test/my_test Then I have entered it into the cron file using crontab -e 0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/test/my_test... (2 Replies)
Discussion started by: Beefy
2 Replies

6. UNIX for Advanced & Expert Users

Cron FTP job

We have a script that we run manually, we want to set it up so that it will run automatically via cron, the problem is during the ftp process nothing is put, the script logs in changes directory fine, but nothing is put to the directory. I have tried it as root, a power user, etc. I have made the... (4 Replies)
Discussion started by: marvin51796
4 Replies

7. UNIX for Dummies Questions & Answers

cron ftp job help

I am new to cron and am trying to set a cron job that will run everyday at 2 to go out to a server (via ftp with user name and password) and get a file and then bring it back to my sever. Below is what I have but it doesn't seem to be working. Any help would be appreciated. cron command - will... (4 Replies)
Discussion started by: bernfuss
4 Replies

8. Shell Programming and Scripting

FTP Cron issues

I am on AS3 Update 4 Linux and am having an issue with an automated ftp script, I tried using the fd/sub proc method and that did not seem to work either. I normally use the following method to perform my ftp's but for some reason it works if I launch the script at the command line but in Cron it... (4 Replies)
Discussion started by: bryanthomas
4 Replies

9. UNIX for Dummies Questions & Answers

ftp in cron job

I am writing a script that will be placed in a crontab, which basically retrieves a file that is overwritten everyday (with current customer data) and then sent to another company. I was wondering what the best way to ensure that ftp was successful and notification of a success or failure message.... (3 Replies)
Discussion started by: walterja
3 Replies

10. UNIX for Advanced & Expert Users

cron issue

user x has a cron job that looks in a dir and moves teh files from 1 name to another except its not working correctly. . /user/.profile # sorce the users profile for file in `ls`; do mv $file $file.`date +%Y%m%d%H%M%S``microsec` done microsec is a binary with 555 perm. on it in... (5 Replies)
Discussion started by: Optimus_P
5 Replies

Featured Tech Videos