Move files from one directory to another in chunks


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
# 1  
Move files from one directory to another in chunks

All,

I have an application that is not working properly and the company is 'in the process' of fixing it. In the meantime, I want to write a bash script work-around. However, what I thought was going to be simple is seemingly not.

Need:

- Move files from one directory to another in chunks.

If I move all the files at once the system resources are heavily consumed and the application suffers. The source directory ($dir1) will contain any number of files (100's or 1000's) at any given time. So, I'd like to do something like:

Code:
if [ "$(ls -A $dir1) ]; then

	find $dir1 -maxdepth 1 -type f > output.txt

	<loop through output.txt moving x amount of files at a time until directory empty>

	mv `head -100 output.txt` $dir2

else

	echo "no files to move.."

fi

I know I can accomplish what I need with a brute force creation of many files to read from but there must be a more sophisticated way to do it.

Is there way to drop the list of files into an array and do something like:

Code:
mv $files[1-100] $dir2
mv $files[101-200] $dir2

I need to find a usable number of files to move at a time (could be 100, 5000, 10000) while not knowing how many files could be in $dir1 at any given time.

Any guidance is greatly appreciated.

TIA,

Herb

Last edited by rbatte1; 08-22-2017 at 07:26 AM.. Reason: Added CODE tags
# 2  
Hi, why dont you try in for loop. with some sleep time enabled. change the SOURCE_DIR and DESTINATION_DIR values according to your source and destination folder
Code:
#!/bin/bash
SOURCE_DIR="/tmp/source"
DESTINATION_DIR="/tmp/destination"

cd ${SOURCE_DIR}
for FileName in *
do
	echo "Moving the File : ${FileName}"
	mv ${FileName} ${DESTINATION_DIR}
	sleep 1
done

This User Gave Thanks to itkamaraj For This Post:
# 3  
You could be simpler with xargs and a call to another mini-script. A starter might be something like:-
Code:
$ cat my_mover.sh
#!/bin/bash
#my_mover.sh script actually does the work

target_dir=$1                 # Grab the target directory
shift                         # Chop off the target directory to just leave all the files listed

mv $@ $target_dir             # Take all arguments and move them to the target directory
sleep 1

You can then run something like this:-
Code:
find $dir1 -maxdepth 1 -type f | xargs -n 100 $dir2 my_mover.sh

This will move 100 files at a time (specified by the xargs parameter after -n)




I hope that this helps,
Robin
This User Gave Thanks to rbatte1 For This Post:
# 4  
Thank you @itkamaraj and @rbatte1 for your replies.

I will test with your suggestions and see what makes the most sense.

Thanks again,

HB
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #543
Difficulty: Easy
A global variable can be accessed and referenced on every line of code.
True or False?

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How Create new directory and move files to that directory.?

Hi All, We have main directory called "head" under this we have several sub directories and under these directories we have sub directories. My requirement is I have to find the SQL files which are having the string "procedure" under "head" directory and sub directories as well. And create... (14 Replies)
Discussion started by: ROCK_PLSQL
14 Replies

2. UNIX for Dummies Questions & Answers

How to move gz files from one source directory to destination directory?

Hi All, Daily i am doing the house keeping in one of my server and manually moving the files which were older than 90 days and moving to destination folder. using the find command . Could you please assist me how to put the automation using the shell script . ... (11 Replies)
Discussion started by: venkat918
11 Replies

3. Shell Programming and Scripting

List files with date, create directory, move to the created directory

Hi all, i have a folder, with tons of files containing as following, on /my/folder/jobs/ some_name_2016-01-17-22-38-58_some name_0_0.zip.done some_name_2016-01-17-22-40-30_some name_0_0.zip.done some_name_2016-01-17-22-48-50_some name_0_0.zip.done and these can be lots of similar files,... (6 Replies)
Discussion started by: charli1
6 Replies

4. Shell Programming and Scripting

Move files in a list to another directory

I have a number of files in a directory that can be grouped with something like "ls | grep SH2". I would like to move each file in this list to another directory. Thanks (4 Replies)
Discussion started by: kg6iia
4 Replies

5. UNIX for Dummies Questions & Answers

Zip all files in a directory and move to another directory

Hi, need to zip all files in a directory and move to another directory after the zip.. i am using this one but didnt help me... zip -r my_proj_`date +%Y%m%d%H%MS`.zip /path/my_proj mv in_proj_`date +%Y%m%d%H%M%S`.zip /path/source/ i am trying to zip all the files in my_proj... (0 Replies)
Discussion started by: dssyadav
0 Replies

6. Shell Programming and Scripting

Move files to variable directory

Im trying to move my files to a variable folder i've just created. Step 1 is working fine. mkdir /sdcard/DCIM/Photos/`date +%Y%m%d` Step 2 is not working mv /sdcard/DCIM/100ANDRO/*.jpg /sdcard/DCIM/Photos/`date +%Y%m%d` What am i doing wrong? (2 Replies)
Discussion started by: dihavs
2 Replies

7. Shell Programming and Scripting

Move files to another directory based on name

Hi Folks, I have different type of file in my current directory. From my current directory i need to move the file which is start with csp_rules and if the file is having the string payg , then I need to move all this files to another directory /output/record. Please help me how to do this? ... (3 Replies)
Discussion started by: suresh01_apk
3 Replies

8. UNIX for Dummies Questions & Answers

How to move all files in a directory and subdirectory?

I'm trying to organize my MB Pro by moving all my jpeg files to a single folder from the desktop. There are some on the desktop that are not in any folder. I was at the command line and typed mv *.jpg "Jpeg files" but it only moved the files that were on the desktop, not any of the ones that... (3 Replies)
Discussion started by: Straitsfan
3 Replies

9. UNIX for Dummies Questions & Answers

Move all files in a directory tree to a signal directory?

Is this possible? Let me know If I need specify further on what I am trying to do- I just want to spare you the boring details of my personal file management. Thanks in advance- Brian- (2 Replies)
Discussion started by: briandanielz
2 Replies

Featured Tech Videos