×
UNIX.COM Login
Username:
Password:  
Show Password






👤


UNIX for Beginners Questions & Answers

If you're not sure where to post a Unix or Linux question, post it here. All unix and Linux beginners welcome in this forum!

Limit number of files transferred

Tags
mget, sftp

👤 Login to reply

 
Thread Tools Search this Thread Display Modes
    #1  
Old 01-02-2018
Janarthan Janarthan is offline
Registered User
 
Join Date: Jan 2018
Last Activity: 19 March 2018, 6:22 AM EDT
Posts: 4
Thanks: 1
Thanked 0 Times in 0 Posts
Limit number of files transferred

I've a folder in remote server and it has 50 files. I like to transfer these files by first 10 and next 10 files.

I'm using mget command to transfer the files. How to limit the file transfer limit to 10. instead of copying 50 files at a time.


Thanks
Janarthan
Sponsored Links
    #2  
Old 01-02-2018
CarloM CarloM is offline Forum Advisor  
Registered User
 
Join Date: Oct 2011
Last Activity: 4 July 2018, 10:21 AM EDT
Location: London, UK
Posts: 1,116
Thanks: 130
Thanked 262 Times in 245 Posts
Using ftp?

Why do you want to do them 10 at a time? Bandwidth issues, or something else?
Sponsored Links
    #3  
Old 01-02-2018
Janarthan Janarthan is offline
Registered User
 
Join Date: Jan 2018
Last Activity: 19 March 2018, 6:22 AM EDT
Posts: 4
Thanks: 1
Thanked 0 Times in 0 Posts
SFTP
in example i mentioned as 10. but in practical I've more than lakhs of files and while doing mget all the files my server memory getting overloaded. To avoid this we're planning to process it by reducing the files count and process it as a batch of 5000 files
    #4  
Old 01-02-2018
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 15 July 2018, 10:19 PM EDT
Location: NM
Posts: 11,435
Thanks: 636
Thanked 1,174 Times in 1,081 Posts
There is a command, parallel, that allows you to perform a selected number of operation all at once. Not all systems have it. Linux generally has it or it can be downloaded and installed.

Poor man's version in bash, using scp to copy files:



Code:
#!/bin/bash
cd /path/to/files/to/send
for fname in *
do
     for (( x=0; x<10; x++ ))
     do
          scp $fname remotebox::  &          
     done
     wait
done

Sponsored Links
    #5  
Old 01-02-2018
Janarthan Janarthan is offline
Registered User
 
Join Date: Jan 2018
Last Activity: 19 March 2018, 6:22 AM EDT
Posts: 4
Thanks: 1
Thanked 0 Times in 0 Posts
If I use parallel then there's a chance for difference in file order sequence between the source and destination path. since, its running parallely
Sponsored Links
    #6  
Old 01-03-2018
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 16 July 2018, 7:55 AM EDT
Location: Aachen, Germany
Posts: 13,059
Thanks: 447
Thanked 4,008 Times in 3,685 Posts
Use a series of sftp batch files (-b option). Get the remote directory listing, split it up into several chunks and write those to the respective batch files.
Sponsored Links
👤 Login to reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
How to check whether files are transferred or not using ftp? Little Shell Programming and Scripting 5 03-26-2013 11:51 PM
Limit Number of files mhs UNIX for Dummies Questions & Answers 1 07-20-2012 08:21 AM
limit number of sub-dirs searched for files MJThom713 UNIX Desktop Questions & Answers 3 07-15-2010 04:24 PM
How to know number files transferred azazalis Shell Programming and Scripting 9 08-13-2006 05:49 AM
limit to number of files in a given directory hedrict UNIX for Advanced & Expert Users 2 02-05-2004 03:14 PM



All times are GMT -4. The time now is 08:40 AM.

Unix & Linux Forums Content Copyrightę1993-2018. All Rights Reserved.