Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

Limit number of files transferred

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Limit number of files transferred
# 1  
Old 01-02-2018
Limit number of files transferred
I've a folder in remote server and it has 50 files. I like to transfer these files by first 10 and next 10 files.

I'm using mget command to transfer the files. How to limit the file transfer limit to 10. instead of copying 50 files at a time.


Thanks
Janarthan
# 2  
Old 01-02-2018
Using ftp?

Why do you want to do them 10 at a time? Bandwidth issues, or something else?
# 3  
Old 01-02-2018
SFTP
in example i mentioned as 10. but in practical I've more than lakhs of files and while doing mget all the files my server memory getting overloaded. To avoid this we're planning to process it by reducing the files count and process it as a batch of 5000 files
# 4  
Old 01-02-2018
There is a command, parallel, that allows you to perform a selected number of operation all at once. Not all systems have it. Linux generally has it or it can be downloaded and installed.

Poor man's version in bash, using scp to copy files:

Code:
#!/bin/bash
cd /path/to/files/to/send
for fname in *
do
     for (( x=0; x<10; x++ ))
     do
          scp $fname remotebox::  &          
     done
     wait
done

# 5  
Old 01-02-2018
If I use parallel then there's a chance for difference in file order sequence between the source and destination path. since, its running parallely
# 6  
Old 01-03-2018
Use a series of sftp batch files (-b option). Get the remote directory listing, split it up into several chunks and write those to the respective batch files.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to check whether files are transferred or not using ftp?

Hi, i want to execute a shell script which transfers files from one server to another using ftp in unix. How can i check whether the ftp is successful or not.(i.e files are transferred to destination server). because if i am checking the return code of ftp, it always shows 0 (denoting ftp is... (5 Replies)
Discussion started by: Little
5 Replies

2. Solaris

How do files transferred via ftp or sftp gets thier permissions at destination?

We have umask defined under /etc/.login as 022. I have my user specific umask defined in /userhome/.login as 002. I understand ftp will not execute anything at destination, it simply transfers files. But it seem to be using 022 as umask for the files transferred. How does ftp knows what umask... (4 Replies)
Discussion started by: kchinnam
4 Replies

3. UNIX for Dummies Questions & Answers

Limit Number of files

hi guys how can i limit number of files in a disk or partition ? or how can i make a limit to inode number for a disk or partition ? ext3 or ext4 file system (1 Reply)
Discussion started by: mhs
1 Replies

4. UNIX Desktop Questions & Answers

limit number of sub-dirs searched for files

using: find . -type f -print|xargs -li "string", how do I limit the dated directories (2010-7-14, 2010-7-13,etc.) to just 2009 & 2010 years of directories to search. We go back to 2004 in our archives, way too many files. (3 Replies)
Discussion started by: MJThom713
3 Replies

5. UNIX for Dummies Questions & Answers

How files can be transferred from one system to another securely using Linux?

i need to know how files can be transfered from one system to another securely in linux. (9 Replies)
Discussion started by: bibing
9 Replies

6. Solaris

How to limit number of Commands

Is there a way that I can limit number of commands that one user can run during period of time. For example Max 10 commands per senconds.:) (3 Replies)
Discussion started by: winger0608
3 Replies

7. UNIX for Dummies Questions & Answers

Number of long limit

Hi Hi! I'm currently using AIX 5.2 and would like to know where can i find to see that there's a restriction on the number of login times a user can have. Example, I want give a 2 login per user but some one to give 3 login and some one have to give unlit login time (without logging off the... (2 Replies)
Discussion started by: herath
2 Replies

8. Shell Programming and Scripting

limit the number of jobs to run.

i just want to get the BackGround status and limit the number of jobs to run the at a time. can i able get the back ground exit status ? that i code below. can jobs -l limit the number of jobs to run ? total is this script looks fine ? cat run_job.ksh #!/usr/bin/ksh... (0 Replies)
Discussion started by: GrepMe
0 Replies

9. Shell Programming and Scripting

How to know number files transferred

Hi, I am transferring files from our local server to remote server using FTP command in a shell script. I am using a the following code, FTPFILE="ercchk*.txt" mput $FTPFILE can any one help me out in calucalating the number of files transfered. help in this regard is highly... (9 Replies)
Discussion started by: azazalis
9 Replies

10. UNIX for Advanced & Expert Users

limit to number of files in a given directory

Everyone, We are on a unix AIX 4.3 platform and our application is written as such that all configuration files must reside in a specific directory. Currently there are over 10,000 files in this directory (and growing at about 300 per month). My question is is there a physical limit to the... (2 Replies)
Discussion started by: hedrict
2 Replies
Login or Register to Ask a Question