Move files from one directory to another in chunks
All,
I have an application that is not working properly and the company is 'in the process' of fixing it. In the meantime, I want to write a bash script work-around. However, what I thought was going to be simple is seemingly not.
Need:
- Move files from one directory to another in chunks.
If I move all the files at once the system resources are heavily consumed and the application suffers. The source directory ($dir1) will contain any number of files (100's or 1000's) at any given time. So, I'd like to do something like:
Code:
if [ "$(ls -A $dir1) ]; then
find $dir1 -maxdepth 1 -type f > output.txt
<loop through output.txt moving x amount of files at a time until directory empty>
mv `head -100 output.txt` $dir2
else
echo "no files to move.."
fi
I know I can accomplish what I need with a brute force creation of many files to read from but there must be a more sophisticated way to do it.
Is there way to drop the list of files into an array and do something like:
Code:
mv $files[1-100] $dir2
mv $files[101-200] $dir2
I need to find a usable number of files to move at a time (could be 100, 5000, 10000) while not knowing how many files could be in $dir1 at any given time.
Any guidance is greatly appreciated.
TIA,
Herb
Last edited by rbatte1; 08-22-2017 at 06:26 AM..
Reason: Added CODE tags
Is this possible? Let me know If I need specify further on what I am trying to do- I just want to spare you the boring details of my personal file management.
Thanks in advance-
Brian- (2 Replies)
Hi All,
I am coding for a requirement where I need to move files (filename.yymmdd) from one directory(A) to another(B) based on 2 date fields in a paramtere file. (Paramfile.txt)
For e.g: In Paramfile.txt,
BUS_DT =20120612
SUB_DT =20120602
In this case, i need to move all the files... (14 Replies)
Hi,
need to zip all files in a directory and move to another directory after the zip..
i am using this one but didnt help me...
zip -r my_proj_`date +%Y%m%d%H%MS`.zip /path/my_proj
mv in_proj_`date +%Y%m%d%H%M%S`.zip /path/source/
i am trying to zip all the files in my_proj... (0 Replies)
I have a directory e2e_ms_xfer/cent01
this contains the multiple files some of which will be named below with unique date time stamps
e2e_ms_edd_nom_CCYYMMDD_HHMM.csv
What I want to do is in a loop
1) Get the oldest file
2) Rename
3) Move it up one level from e2e_ms_xfer/cent01 to... (1 Reply)
I have a directory e2e_ms_xfer/cent01
this contains the multiple files some of which will be named below with unique date time stamps
e2e_ms_edd_nom_CCYYMMDD_HHMM.csv
What I want to do is in a loop
1) Get the oldest file
2) Rename
3) Move it up one level from e2e_ms_xfer/cent01 to... (1 Reply)
Hi all,
i have a folder, with tons of files containing as following,
on /my/folder/jobs/
some_name_2016-01-17-22-38-58_some name_0_0.zip.done
some_name_2016-01-17-22-40-30_some name_0_0.zip.done
some_name_2016-01-17-22-48-50_some name_0_0.zip.done
and these can be lots of similar files,... (6 Replies)
I want to move all the files in a given directory up one level.
For example:
Dir1
Subdir1
I want to move all the files in Subdir1 up to Dir1 (then I want to ultimately delete Subdir1)
Thanks,
Ted (10 Replies)
Hi All,
Daily i am doing the house keeping in one of my server and manually moving the files which were older than 90 days and moving to destination folder.
using the find command . Could you please assist me how to put the automation using the shell script .
... (11 Replies)
Hi All,
We have main directory called "head"
under this we have several sub directories and under these directories we have sub directories.
My requirement is I have to find the SQL files which are having the string "procedure" under "head" directory and sub directories as well.
And create... (14 Replies)
Discussion started by: ROCK_PLSQL
14 Replies
LEARN ABOUT DEBIAN
rsnapshot-diff
rsnapshot-diff(1)rsnapshot-diff(1)NAME
rsnapshot-diff - a utility for comparing the disk usage of two snapshots taken by rsnapshot
SYNOPSIS
rsnapshot-diff [-h|vVi] dir1 dir2
DESCRIPTION
rsnapshot-diff is a companion utility for rsnapshot, which traverses two parallel directory structures and calculates the difference
between them. By default it is silent apart from displaying summary information at the end, but it can be made more verbose.
In the summary, "added" files may very well include files which at first glance also appear at the same place in the older directory
structure. However, because the files differ in some respect, they are different files. They have a different inode number. Consequently
if you use -v most of its output may appear to be pairs of files with the same name being removed and added.
OPTIONS -h (help)
Displays help information
-v (verbose)
Be verbose. This will spit out a list of all changes as they are encountered, as well as the summary at the end.
-V (more verbose)
Be more verbose - as well as listed changes, unchanged files will be listed too.
-i (ignore)
If verbosity is turned on, -i suppresses information about symlinks, directories, and special files.
dir1 and dir2
These are the only compulsory parameters, and should be the names of two directories to compare. Their order doesn't matter,
rsnapshot-diff will always compare the younger to the older, so files that appear only in the older will be reported as having been
removed, and files that appear only in the younger will be reported as having been added.
SEE ALSO
rsnapshot
BUGS
Please report bugs (and other comments) to the rsnapshot-discuss mailing list:
http://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss <http://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss>
AUTHOR
David Cantrell <david@cantrell.org.uk>
COPYRIGHT
Copyright 2005 David Cantrell
LICENCE
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation,
Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
2011-07-09 rsnapshot-diff(1)