Home Man
Search
Today's Posts
Register

If you're not sure where to post a Unix or Linux question, post it here. All unix and Linux beginners welcome in this forum!

Limit number of files transferred

Tags
mget, sftp

Login to Reply

 
Thread Tools Search this Thread
# 1  
Old 01-02-2018
Limit number of files transferred

I've a folder in remote server and it has 50 files. I like to transfer these files by first 10 and next 10 files.

I'm using mget command to transfer the files. How to limit the file transfer limit to 10. instead of copying 50 files at a time.


Thanks
Janarthan
# 2  
Old 01-02-2018
Using ftp?

Why do you want to do them 10 at a time? Bandwidth issues, or something else?
# 3  
Old 01-02-2018
SFTP
in example i mentioned as 10. but in practical I've more than lakhs of files and while doing mget all the files my server memory getting overloaded. To avoid this we're planning to process it by reducing the files count and process it as a batch of 5000 files
# 4  
Old 01-02-2018
There is a command, parallel, that allows you to perform a selected number of operation all at once. Not all systems have it. Linux generally has it or it can be downloaded and installed.

Poor man's version in bash, using scp to copy files:

Code:
#!/bin/bash
cd /path/to/files/to/send
for fname in *
do
     for (( x=0; x<10; x++ ))
     do
          scp $fname remotebox::  &          
     done
     wait
done

# 5  
Old 01-02-2018
If I use parallel then there's a chance for difference in file order sequence between the source and destination path. since, its running parallely
# 6  
Old 01-03-2018
Use a series of sftp batch files (-b option). Get the remote directory listing, split it up into several chunks and write those to the respective batch files.
Login to Reply

« Previous Thread | Next Thread »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
How to check whether files are transferred or not using ftp? Little Shell Programming and Scripting 5 03-26-2013 11:51 PM
How do files transferred via ftp or sftp gets thier permissions at destination? kchinnam Solaris 4 01-10-2013 08:42 AM
Limit Number of files mhs UNIX for Dummies Questions & Answers 1 07-20-2012 08:21 AM
limit number of sub-dirs searched for files MJThom713 UNIX Desktop Questions & Answers 3 07-15-2010 04:24 PM
How files can be transferred from one system to another securely using Linux? bibing UNIX for Dummies Questions & Answers 9 03-03-2010 04:03 PM
How to limit number of Commands winger0608 Solaris 3 06-30-2009 11:25 AM
Number of long limit herath UNIX for Dummies Questions & Answers 2 06-03-2008 02:31 AM
Limit of number of lines for awk and sed. 60doses Shell Programming and Scripting 2 01-17-2008 04:52 AM
How to know number files transferred azazalis Shell Programming and Scripting 9 08-13-2006 05:49 AM
limit to number of files in a given directory hedrict UNIX for Advanced & Expert Users 2 02-05-2004 03:14 PM


All times are GMT -4. The time now is 01:50 AM.

Unix & Linux Forums Content Copyrightę1993-2018. All Rights Reserved.
UNIX.COM Login
Username:
Password:  
Show Password