Script to move certain no. of files every second


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to move certain no. of files every second
# 1  
Old 08-03-2017
Linux Script to move certain no. of files every second

Hi All,

I am new to Linux/Scripting and need some assistance in coming up with a script that can move certain amount of files from one directory to other every seconds.

Usercase: We have around 100k files in tmp directory on my server which needs to be moved to another folder to get processed. However I can not move all files at once due to limitations with file parsing as it hangs after seeing 100k files at once and can move around 10 files per seconds.

It would be helpful if someone can suggest a best way to achieve this through a shell script.

Thanks in advance.


Regards,
Raj
# 2  
Old 08-03-2017
How about renaming the directory?
# 3  
Old 08-03-2017
renaming wouldn't help in my case.

Thank you for your response
# 4  
Old 08-03-2017
Why do you care how many files you can move in a second?

Why not just start moving files from the 1st file you want to move to the last file you want to move and just quit when you're done?

What are your "limitations with file parsing"? You can easily run into maximum argument length limits if you are trying to construct an mv command with 10,000 pathnames as operands; but that isn't file parsing. And, it can easily be worked around with the find utility with a couple of selection primaries and an -exec primary to perform the moves in chunk that will not exceed the arglist length limits on your system.

Depending on what operating system and shell you're using there may also be other ways to do what you need to do; but with the very limited information you have provided, it is very hard to make any reasonable suggestions.

If it really matters, note that trying to guess how many files can be moved in a second varies greatly from operating system to operating system, from filesystem type to filesystem type, how much activity there is on the filesystems involved, and on network characteristics if you're trying to move files to a remote directory as well as depending on the hardware being used and the sizes of the files being moved. You haven't given us any of the information that would be required to meet your requirements
Login or Register to Ask a Question

Previous Thread | Next Thread

7 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script move files by name

hi, I have a lot of files named xxxxx__AA.txt, xxxxx__BB.txt, xxxxx__CC.txt and I would like to move xxxxx__AA.txt in AA directory, xxxxx__BB.txt in BB etc. Could you help me do it in bash script? (5 Replies)
Discussion started by: corfuitl
5 Replies

2. Shell Programming and Scripting

help with a script to gzip/move files

Hi Please can you help me in writing a script to find files on a specific directory, and of extension "tap" but only of the month of september, gzip and move them to another directory. Your help will be appreciated. (4 Replies)
Discussion started by: fretagi
4 Replies

3. Shell Programming and Scripting

Need shell script to move files

Hi , I need a simple shell script to move the files from one directory to another directory after every 1 hour..!!! ?? (1 Reply)
Discussion started by: SARAL SAXENA
1 Replies

4. Shell Programming and Scripting

Need script to move files based on name

Hi folks, I'm new here and appreciate greatly any help. I have a bunch of files that need be moved and renamed. Fortunately, they are all in sequence... Present filename and path: /.catalog1/t76038_842-01 Move to: /.catalog1/76038-01 So, we need to drop the... (8 Replies)
Discussion started by: axslinger
8 Replies

5. Shell Programming and Scripting

Perl script to move files not in use

I need to write a script to move files only when they are not in use. I have a rudementry bash script for Linux but i need a perl script so it will work on Linux and hpux. Oracle writes files to a directory called /data and the files there are moved every 5 minutes to a new home. But i need to... (1 Reply)
Discussion started by: insania
1 Replies

6. UNIX for Dummies Questions & Answers

Script to move blank files

Anyone could give me an example of scrip to move blank files found into a dir? Thanks, Leandro Takeda (3 Replies)
Discussion started by: letakeda
3 Replies

7. Shell Programming and Scripting

Script to move trace files

Please debug this shell script for me.. Basically the idea is to run the script, based on the command to move some trace files to a separate directory and I am getting the error. Only the COMMAND that has rm {} works and I basically want to use it for the fourth one. Please try for the 2nd, 3rd and... (4 Replies)
Discussion started by: ST2000
4 Replies
Login or Register to Ask a Question