trnsmiting thousands ftp files and get an error message | Unix Linux Forums | Shell Programming and Scripting

  Go Back    


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

trnsmiting thousands ftp files and get an error message

Shell Programming and Scripting


Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 09-27-2006
alexcol alexcol is offline
Registered User
 
Join Date: Sep 2006
Last Activity: 21 August 2014, 12:39 PM EDT
Location: Bogota - Colombia - South America
Posts: 58
Thanks: 0
Thanked 1 Time in 1 Post
Data trnsmiting thousands ftp files and get an error message

Im transmiting thousands ftp files to a server, when type the command mput *, an error comes and say. args list to long. set to I. So ihave to transmit them in batch or blocks, but its too sloww. what shoul i do?. i need to do a program, or with a simple command i could solve the problem?

Last edited by alexcol; 09-27-2006 at 02:00 PM..
Sponsored Links
    #2  
Old 09-27-2006
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 21 August 2014, 1:38 PM EDT
Location: Saskatchewan
Posts: 19,231
Thanks: 773
Thanked 3,223 Times in 3,021 Posts
"mput *" is expanded in the shell, before it is executed, to "mput file1 file2 file3 ..." There is generally a limit of 32K or so for how long the line can be, which is why you get this error when trying to match * for thousands of files.

I'm not familiar with mput. Is it possible for it to take a list of files instead of arguments on the commandline? You could just do "ls > /tmp/filelist" to make the list.

You can also use xargs to split down that monolilthic list into more manageable batches. Keep the batches large enough and it shouldn't be too much slower. Try this:

Code:
# List files in the current directory, piping the output into xargs
ls ./ |
# For each batch of 100 or less, execute "mput file1 file2 ... filen" where filen is the nth file name in the batch.
    xargs --max-args=100 mput

ls will list the files one per line, xargs will group them together in batches of 100 and call 'mput file1 file2 file3 ... file100' for each batch.
Sponsored Links
    #3  
Old 09-27-2006
Andrek Andrek is offline
Registered User
 
Join Date: Aug 2006
Last Activity: 27 April 2010, 9:08 PM EDT
Posts: 122
Thanks: 0
Thanked 0 Times in 0 Posts
Since your "mput'ing" you have access to the source files.
What if you used tar to save all the files to a single tar file and them ftp the single tar file over.....
You could create smaller tar files if the size is to big.

Other options for remote transfer are rcp or scp.

Or nfs mount the file system and copy the files directy.
    #4  
Old 09-27-2006
reborg's Avatar
reborg reborg is offline Forum Advisor  
Administrator Emeritus
 
Join Date: Mar 2005
Last Activity: 29 March 2012, 7:00 PM EDT
Location: Ireland
Posts: 4,464
Thanks: 0
Thanked 11 Times in 11 Posts
or do it on the fly with

put "| tar cf - ." filename.tar
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
help to parallelize work on thousands of files vhope07 Shell Programming and Scripting 10 07-12-2010 06:53 AM
Copying Thousands of Tiny or Empty Files? deckard UNIX for Advanced & Expert Users 3 04-28-2009 03:51 PM
compare two files if doesnt match then display error message atl@mav UNIX for Dummies Questions & Answers 2 03-03-2009 09:57 AM
Finding a specific pattern from thousands of files ???? aarora_98 Shell Programming and Scripting 6 02-17-2006 07:28 AM
Suppres error message when moving files from empty source folder Steven Shell Programming and Scripting 2 11-19-2001 12:25 PM



All times are GMT -4. The time now is 03:59 PM.