The UNIX and Linux Forums  

Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
Google UNIX.COM


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Unix Arithmatic operation issue , datatype issue thambi Shell Programming and Scripting 23 02-19-2008 03:19 AM
Built in ram limit for 64 bit kermit Linux 2 05-18-2007 12:42 PM
tar 2GB limit SLKRR Filesystems, Disks and Memory 9 12-04-2006 11:19 AM
Limit logins to 1 pheusion AIX 0 08-11-2006 08:23 AM
Limit command macdonto UNIX for Dummies Questions & Answers 1 10-21-2002 12:58 PM

Reply
 
Submit Tools LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 01-22-2004
whegra's Avatar
Registered User
 

Join Date: Sep 2002
Posts: 19
Question Ftp Limit issue

I regularly have to upload files from my Unix server to a windows box. However I'm running into and issue ftp seems to have a limit to the amount of file that it can upload in 1 shot.

What I mean is, say I have 10000 files that start with E

If I try to upload all files via mput E* is get "Arguments too long"

This is causing a big issue for me as I have everything automated via some scripts that I wrote.

I was thinking of some how doing a list then having ftp read that list and upload file by file not via mput e*, but I'm not really sure how to pass a list of files to ftp.

I also looked a Perderabo script that he wrote, but it transfers in the wrong direction for me.

I searched for an answer to this but couldn't find anything, if I missed something, please point me in the direction of that thread.

Any help would be great

Thank you.
Reply With Quote
Forum Sponsor
  #2 (permalink)  
Old 01-22-2004
oombera's Avatar
Have a day :|
 

Join Date: Aug 2002
Location: Cleveland, OH
Posts: 804
This sounds like what you want.. at least the question is the same. The answer, however, I did not test, so basically I'm just throwing a suggestion out there..

http://www.landfield.com/wu-ftpd/mai.../Oct/0013.html
Quote:
Sounds like an internal buffer limitation in the FTP client.

I'd write a shell script to make a file and pipe it into ftp. I do this
by-hand when I'm updating a give repeatedly. Consider:

(echo "bin" ; echo "put master.c" ; echo "quit" ) | ftp ftp.example.com

If I have my username and password in .netrc the login is automatic, the piped commands fire off and I get a command prompt when the transfer's complete. If I have a complex problem, or need to see the ftp output, I use expect to script the ftp client.

Last edited by oombera; 01-22-2004 at 08:19 AM.
Reply With Quote
  #3 (permalink)  
Old 01-22-2004
Perderabo's Avatar
Unix Daemon
 

Join Date: Aug 2001
Location: Washington DC Area
Posts: 8,466
A quick fix is to go to the 2nd or 3rd character.

mput E01*
mput E02*
mput E03*
etc
Reply With Quote
  #4 (permalink)  
Old 01-22-2004
whegra's Avatar
Registered User
 

Join Date: Sep 2002
Posts: 19
Quote:
(echo "bin" ; echo "put master.c" ; echo "quit" ) | ftp ftp.example.com
I understand some of this bin = change to binary mode and quit = quit the ftp session

but what is put master.c is it just upload that single file or does it upload all the files listed in master.c ? to me it looks like it just uploads the single file.

Quote:
A quick fix is to go to the 2nd or 3rd character.

mput E01*
mput E02*
mput E03*
etc
I'm currently doing this but the files arn't always E's it varies day by day, and is creating a lot of extra work for me.

Is there not a way to pipe the output of directory listing to a file and then in turn pipe that to the ftp program so that it will go file by file instead of having to the mput e* or e01* in my script.

Thanks for the help guys.
Reply With Quote
  #5 (permalink)  
Old 01-22-2004
flim flam flamma jamma
 

Join Date: May 2001
Location: Chicago IL, USA
Posts: 1,006
do a search for "Arguments too long" on this site.

It was explaned why this happens.
its a limitation on the way the mput gets a list of its arguments.

the quick fix is to tar up all the files then just send the tar file to your windows machine.

OR cut your process up to putting 1024 files at a time.

The below code is untested but it should work.

Code:
#!/usr/bin/perl -w

use strict;
use Net::FTP;

my ($host, $user, $pass, $localfs, $remotefs)=( 'your_host', 'ur_username', 'ur_passwd', 'dir_holding_files' 'dir_2_upload_files);
chdir("$localfs") or die "can not change directory /homes($!)";;
my @ftp_files=glob("*");
$ftp = Net::FTP->new("$host", Debug => 0 ) or warn "Cannot connect to $host: $@";
$ftp->login("$user","$pass") or warn "Cannot login ", $ftp->message;
$ftp->binary();
$ftp->cwd("$remotefs");
for (@ftp_files) { $ftp->put("$_") };
$ftp->quit;

Last edited by Optimus_P; 01-22-2004 at 09:42 AM.
Reply With Quote
  #6 (permalink)  
Old 01-22-2004
whegra's Avatar
Registered User
 

Join Date: Sep 2002
Posts: 19
Optimus_P, thanks for the code, 1 small issue, my unix server which is SCO OpenServer Enterprise System (ver 5.0.5m) doesn't have perl unfortunitly.
Reply With Quote
  #7 (permalink)  
Old 01-22-2004
Perderabo's Avatar
Unix Daemon
 

Join Date: Aug 2001
Location: Washington DC Area
Posts: 8,466
Well. it's a slow day. So
Code:
#! /usr/bin/ksh

stty -echo
print -n Enter Password-
read PASS
print 
stty echo
exec 4>&1

HOST=xxx.yyy.com
USER=name
DESTDIR=junk
SOURCEDIR=~/tmp

cd $SOURCEDIR

ftp -nv >&4 2>&4 |&
print -p open $HOST
print -p user $USER $PASS
print -p cd $DESTDIR

ls | while read filename ; do
        [[ -f $filename ]] && print -p put $filename
done
print -p close
print -p bye
wait
exit 0
I hope you have ksh.
Reply With Quote
Google The UNIX and Linux Forums
Reply

Thread Tools
Display Modes




All times are GMT -7. The time now is 01:22 PM.


Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited.
The UNIX and Linux Forums Content Copyright ©1993-2008. All Rights Reserved.Ad Management by RedTyger Visit The Global Fact Book

Content Relevant URLs by vBSEO 3.2.0