Unix/Linux Go Back    


UNIX for Beginners Questions & Answers If you're not sure where to post a Unix or Linux question, post it here. All unix and Linux beginners welcome in this forum!

Limit number of files transferred

UNIX for Beginners Questions & Answers


Tags
mget, sftp

Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 01-02-2018   -   Original Discussion by Janarthan
Janarthan's Unix or Linux Image
Janarthan Janarthan is offline
Registered User
 
Join Date: Jan 2018
Last Activity: 19 March 2018, 6:22 AM EDT
Posts: 4
Thanks: 1
Thanked 0 Times in 0 Posts
Limit number of files transferred

I've a folder in remote server and it has 50 files. I like to transfer these files by first 10 and next 10 files.

I'm using mget command to transfer the files. How to limit the file transfer limit to 10. instead of copying 50 files at a time.


Thanks
Janarthan
Sponsored Links
    #2  
Old Unix and Linux 01-02-2018   -   Original Discussion by Janarthan
CarloM's Unix or Linux Image
CarloM CarloM is offline Forum Advisor  
Registered User
 
Join Date: Oct 2011
Last Activity: 24 May 2018, 7:16 AM EDT
Location: London, UK
Posts: 1,115
Thanks: 129
Thanked 262 Times in 245 Posts
Using ftp?

Why do you want to do them 10 at a time? Bandwidth issues, or something else?
Sponsored Links
    #3  
Old Unix and Linux 01-02-2018   -   Original Discussion by Janarthan
Janarthan's Unix or Linux Image
Janarthan Janarthan is offline
Registered User
 
Join Date: Jan 2018
Last Activity: 19 March 2018, 6:22 AM EDT
Posts: 4
Thanks: 1
Thanked 0 Times in 0 Posts
SFTP
in example i mentioned as 10. but in practical I've more than lakhs of files and while doing mget all the files my server memory getting overloaded. To avoid this we're planning to process it by reducing the files count and process it as a batch of 5000 files
    #4  
Old Unix and Linux 01-02-2018   -   Original Discussion by Janarthan
jim mcnamara's Unix or Linux Image
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 27 May 2018, 9:24 AM EDT
Location: NM
Posts: 11,393
Thanks: 616
Thanked 1,161 Times in 1,068 Posts
There is a command, parallel, that allows you to perform a selected number of operation all at once. Not all systems have it. Linux generally has it or it can be downloaded and installed.

Poor man's version in bash, using scp to copy files:



Code:
#!/bin/bash
cd /path/to/files/to/send
for fname in *
do
     for (( x=0; x<10; x++ ))
     do
          scp $fname remotebox::  &          
     done
     wait
done

Sponsored Links
    #5  
Old Unix and Linux 01-02-2018   -   Original Discussion by Janarthan
Janarthan's Unix or Linux Image
Janarthan Janarthan is offline
Registered User
 
Join Date: Jan 2018
Last Activity: 19 March 2018, 6:22 AM EDT
Posts: 4
Thanks: 1
Thanked 0 Times in 0 Posts
If I use parallel then there's a chance for difference in file order sequence between the source and destination path. since, its running parallely
Sponsored Links
    #6  
Old Unix and Linux 01-03-2018   -   Original Discussion by Janarthan
RudiC's Unix or Linux Image
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 26 May 2018, 5:24 PM EDT
Location: Aachen, Germany
Posts: 12,756
Thanks: 417
Thanked 3,923 Times in 3,607 Posts
Use a series of sftp batch files (-b option). Get the remote directory listing, split it up into several chunks and write those to the respective batch files.
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
How to check whether files are transferred or not using ftp? Little Shell Programming and Scripting 5 03-26-2013 11:51 PM
Limit Number of files mhs UNIX for Dummies Questions & Answers 1 07-20-2012 08:21 AM
limit number of sub-dirs searched for files MJThom713 UNIX Desktop Questions & Answers 3 07-15-2010 04:24 PM
How to know number files transferred azazalis Shell Programming and Scripting 9 08-13-2006 05:49 AM
limit to number of files in a given directory hedrict UNIX for Advanced & Expert Users 2 02-05-2004 03:14 PM



All times are GMT -4. The time now is 11:51 AM.