Unix/Linux Go Back    

Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

trnsmiting thousands ftp files and get an error message

Shell Programming and Scripting

Thread Tools Search this Thread Display Modes
Old Unix and Linux 09-27-2006
alexcol alexcol is offline
Registered User
Join Date: Sep 2006
Last Activity: 16 October 2017, 5:17 PM EDT
Location: Bogota - Colombia - South America
Posts: 97
Thanks: 3
Thanked 1 Time in 1 Post
Data trnsmiting thousands ftp files and get an error message

Im transmiting thousands ftp files to a server, when type the command mput *, an error comes and say. args list to long. set to I. So ihave to transmit them in batch or blocks, but its too sloww. what shoul i do?. i need to do a program, or with a simple command i could solve the problem?

Last edited by alexcol; 09-27-2006 at 02:00 PM..
Sponsored Links
Old Unix and Linux 09-27-2006
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
Join Date: Aug 2005
Last Activity: 20 October 2017, 3:09 PM EDT
Location: Saskatchewan
Posts: 22,469
Thanks: 1,143
Thanked 4,262 Times in 3,936 Posts
"mput *" is expanded in the shell, before it is executed, to "mput file1 file2 file3 ..." There is generally a limit of 32K or so for how long the line can be, which is why you get this error when trying to match * for thousands of files.

I'm not familiar with mput. Is it possible for it to take a list of files instead of arguments on the commandline? You could just do "ls > /tmp/filelist" to make the list.

You can also use xargs to split down that monolilthic list into more manageable batches. Keep the batches large enough and it shouldn't be too much slower. Try this:

# List files in the current directory, piping the output into xargs
ls ./ |
# For each batch of 100 or less, execute "mput file1 file2 ... filen" where filen is the nth file name in the batch.
    xargs --max-args=100 mput

ls will list the files one per line, xargs will group them together in batches of 100 and call 'mput file1 file2 file3 ... file100' for each batch.
Sponsored Links
Old Unix and Linux 09-27-2006
Andrek Andrek is offline
Registered User
Join Date: Aug 2006
Last Activity: 27 April 2010, 9:08 PM EDT
Posts: 122
Thanks: 0
Thanked 0 Times in 0 Posts
Since your "mput'ing" you have access to the source files.
What if you used tar to save all the files to a single tar file and them ftp the single tar file over.....
You could create smaller tar files if the size is to big.

Other options for remote transfer are rcp or scp.

Or nfs mount the file system and copy the files directy.
Old Unix and Linux 09-27-2006
reborg's Unix or Linux Image
reborg reborg is offline Forum Advisor  
Administrator Emeritus
Join Date: Mar 2005
Last Activity: 29 March 2012, 7:00 PM EDT
Location: Ireland
Posts: 4,464
Thanks: 0
Thanked 13 Times in 11 Posts
or do it on the fly with

put "| tar cf - ." filename.tar
Sponsored Links

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
help to parallelize work on thousands of files vhope07 Shell Programming and Scripting 10 07-12-2010 06:53 AM
Copying Thousands of Tiny or Empty Files? deckard UNIX for Advanced & Expert Users 3 04-28-2009 03:51 PM
compare two files if doesnt match then display error message atl@mav UNIX for Dummies Questions & Answers 2 03-03-2009 09:57 AM
Finding a specific pattern from thousands of files ???? aarora_98 Shell Programming and Scripting 6 02-17-2006 07:28 AM
Suppres error message when moving files from empty source folder Steven Shell Programming and Scripting 2 11-19-2001 12:25 PM

All times are GMT -4. The time now is 11:15 PM.