Limit number of files transferred

mget, sftp

Thread Tools Search this Thread
# 1  
Old 01-02-2018
Limit number of files transferred

I've a folder in remote server and it has 50 files. I like to transfer these files by first 10 and next 10 files.

I'm using mget command to transfer the files. How to limit the file transfer limit to 10. instead of copying 50 files at a time.

# 2  
Old 01-02-2018
Using ftp?

Why do you want to do them 10 at a time? Bandwidth issues, or something else?
# 3  
Old 01-02-2018
in example i mentioned as 10. but in practical I've more than lakhs of files and while doing mget all the files my server memory getting overloaded. To avoid this we're planning to process it by reducing the files count and process it as a batch of 5000 files
# 4  
Old 01-02-2018
There is a command, parallel, that allows you to perform a selected number of operation all at once. Not all systems have it. Linux generally has it or it can be downloaded and installed.

Poor man's version in bash, using scp to copy files:

cd /path/to/files/to/send
for fname in *
     for (( x=0; x<10; x++ ))
          scp $fname remotebox::  &          

# 5  
Old 01-02-2018
If I use parallel then there's a chance for difference in file order sequence between the source and destination path. since, its running parallely
# 6  
Old 01-03-2018
Use a series of sftp batch files (-b option). Get the remote directory listing, split it up into several chunks and write those to the respective batch files.

Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Is there limit on number of ACLs' per directory in Redhat waavman Red Hat 2 03-02-2017 09:52 AM
How to check whether files are transferred or not using ftp? Little Shell Programming and Scripting 5 03-27-2013 12:51 AM
How do files transferred via ftp or sftp gets thier permissions at destination? kchinnam Solaris 4 01-10-2013 09:42 AM
Reading from a file with limit number of lines krux_rap Shell Programming and Scripting 3 11-05-2012 08:10 PM
Limit Number of files mhs UNIX for Dummies Questions & Answers 1 07-20-2012 09:21 AM
Limit the number of characters in a bash output Light_ UNIX for Dummies Questions & Answers 2 10-22-2011 10:18 PM
limit number of sub-dirs searched for files MJThom713 UNIX Desktop Questions & Answers 3 07-15-2010 05:24 PM
How files can be transferred from one system to another securely using Linux? bibing UNIX for Dummies Questions & Answers 9 03-03-2010 05:03 PM
How to limit the number of child processes xyzt Programming 3 11-14-2009 04:15 PM
How to limit number of Commands winger0608 Solaris 3 06-30-2009 12:25 PM
Number of long limit herath UNIX for Dummies Questions & Answers 2 06-03-2008 03:31 AM
Limit of number of lines for awk and sed. 60doses Shell Programming and Scripting 2 01-17-2008 05:52 AM
limit the number of jobs to run. GrepMe Shell Programming and Scripting 0 06-15-2007 02:39 PM
How to know number files transferred azazalis Shell Programming and Scripting 9 08-13-2006 06:49 AM
limit to number of files in a given directory hedrict UNIX for Advanced & Expert Users 2 02-05-2004 04:14 PM