Move files from one directory to another in chunks
All,
I have an application that is not working properly and the company is 'in the process' of fixing it. In the meantime, I want to write a bash script work-around. However, what I thought was going to be simple is seemingly not.
Need:
- Move files from one directory to another in chunks.
If I move all the files at once the system resources are heavily consumed and the application suffers. The source directory ($dir1) will contain any number of files (100's or 1000's) at any given time. So, I'd like to do something like:
Code:
if [ "$(ls -A $dir1) ]; then
find $dir1 -maxdepth 1 -type f > output.txt
<loop through output.txt moving x amount of files at a time until directory empty>
mv `head -100 output.txt` $dir2
else
echo "no files to move.."
fi
I know I can accomplish what I need with a brute force creation of many files to read from but there must be a more sophisticated way to do it.
Is there way to drop the list of files into an array and do something like:
Code:
mv $files[1-100] $dir2
mv $files[101-200] $dir2
I need to find a usable number of files to move at a time (could be 100, 5000, 10000) while not knowing how many files could be in $dir1 at any given time.
Any guidance is greatly appreciated.
TIA,
Herb
Last edited by rbatte1; 08-22-2017 at 06:26 AM..
Reason: Added CODE tags
Is this possible? Let me know If I need specify further on what I am trying to do- I just want to spare you the boring details of my personal file management.
Thanks in advance-
Brian- (2 Replies)
Hi All,
I am coding for a requirement where I need to move files (filename.yymmdd) from one directory(A) to another(B) based on 2 date fields in a paramtere file. (Paramfile.txt)
For e.g: In Paramfile.txt,
BUS_DT =20120612
SUB_DT =20120602
In this case, i need to move all the files... (14 Replies)
Hi,
need to zip all files in a directory and move to another directory after the zip..
i am using this one but didnt help me...
zip -r my_proj_`date +%Y%m%d%H%MS`.zip /path/my_proj
mv in_proj_`date +%Y%m%d%H%M%S`.zip /path/source/
i am trying to zip all the files in my_proj... (0 Replies)
I have a directory e2e_ms_xfer/cent01
this contains the multiple files some of which will be named below with unique date time stamps
e2e_ms_edd_nom_CCYYMMDD_HHMM.csv
What I want to do is in a loop
1) Get the oldest file
2) Rename
3) Move it up one level from e2e_ms_xfer/cent01 to... (1 Reply)
I have a directory e2e_ms_xfer/cent01
this contains the multiple files some of which will be named below with unique date time stamps
e2e_ms_edd_nom_CCYYMMDD_HHMM.csv
What I want to do is in a loop
1) Get the oldest file
2) Rename
3) Move it up one level from e2e_ms_xfer/cent01 to... (1 Reply)
Hi all,
i have a folder, with tons of files containing as following,
on /my/folder/jobs/
some_name_2016-01-17-22-38-58_some name_0_0.zip.done
some_name_2016-01-17-22-40-30_some name_0_0.zip.done
some_name_2016-01-17-22-48-50_some name_0_0.zip.done
and these can be lots of similar files,... (6 Replies)
I want to move all the files in a given directory up one level.
For example:
Dir1
Subdir1
I want to move all the files in Subdir1 up to Dir1 (then I want to ultimately delete Subdir1)
Thanks,
Ted (10 Replies)
Hi All,
Daily i am doing the house keeping in one of my server and manually moving the files which were older than 90 days and moving to destination folder.
using the find command . Could you please assist me how to put the automation using the shell script .
... (11 Replies)
Hi All,
We have main directory called "head"
under this we have several sub directories and under these directories we have sub directories.
My requirement is I have to find the SQL files which are having the string "procedure" under "head" directory and sub directories as well.
And create... (14 Replies)
Discussion started by: ROCK_PLSQL
14 Replies
LEARN ABOUT MINIX
synctree
SYNCTREE(1) General Commands Manual SYNCTREE(1)NAME
synctree - synchronize directory trees.
SYNOPSIS
synctree [-iuf] [[user1@]machine1:]dir1 [[user2@]machine2:]dir2
DESCRIPTION
Synctree synchronizes the directory tree rooted at dir2 with dir1. It walks recursively through both trees, and deletes and adds files in
dir2 to make it equal to dir1. Mode, owner and group are set for each file unless the -u flag is given. In its normal mode of operation,
synctree will ask if it may delete or add directories assuming that you don't want to. Non-directories are simply deleted or added, but
synctree will ask if it needs to update a normal file with a default answer of 'y'. Simply typing return will choose the default answer,
typing end-of-file is like typing return to this question and all other questions.
You can specify a hostname and user-id to be used to access dir1 or dir2. Synctree will use rsh(1) to run a copy of itself on the remote
machine. The call interface mimics that of rcp(1), but you can use more than one user@machine prefix if you want to make things really
interesting.
Hard links are enforced, an update is done by first deleting the old file so that links to unknown files are broken. Links to files within
dir2 will be restored.
If either directory contains the file .backup, then this file will be used as an alternate inode table. This allows one to make a backup
copy of a file tree full of special files and differing user-ids on a remote machine under an unpriviledged user-id.
OPTIONS -i Ask for permission (with default answer 'n') to delete or add any file or directory.
-u Only install newer files, i.e. merge the directory trees.
-f Don't ask, think 'yes' on any question.
SEE ALSO remsync(1), cpdir(1), rsh(1), rcp(1), perror(3).
DIAGNOSTICS
Messages may come from three different processes. One named "Slave" running in dir1, one named "Master" running in dir2, and synctree
itself in a mediator role. The mediator will also perform the task of either the master or the slave if one of them is running locally.
You need to know this to interpret the error messages coming from one of these processes. The messages are normally based on perror(3).
Failure to contact a remote machine will be reported by rsh. Synctree should have a zero exit status if no errors have been encountered.
BUGS
Directory dir2 will be created without asking.
The master and slave processes get their error output mixed up sometimes (nice puzzle).
The local and remote machine must use the same file type encoding.
The link replacement strategy may lead to lack of space on a small device. Let synctree run to completion and then rerun it to pick up the
pieces.
Letting the local process keep its "synctree" name may be a mistake.
It talks too much.
AUTHOR
Kees J. Bot, (kjb@cs.vu.nl)
SYNCTREE(1)