Move files from one directory to another in chunks Post: 303002302

Sponsored Content

Top Forums Shell Programming and Scripting Move files from one directory to another in chunks Post 303002302 by hburnswell on Monday 21st of August 2017 11:46:49 PM

08-22-2017

Registered User

Move files from one directory to another in chunks

All,

I have an application that is not working properly and the company is 'in the process' of fixing it. In the meantime, I want to write a bash script work-around. However, what I thought was going to be simple is seemingly not.

Need:

- Move files from one directory to another in chunks.

If I move all the files at once the system resources are heavily consumed and the application suffers. The source directory ($dir1) will contain any number of files (100's or 1000's) at any given time. So, I'd like to do something like:

Code:

if [ "$(ls -A $dir1) ]; then

	find $dir1 -maxdepth 1 -type f > output.txt

	<loop through output.txt moving x amount of files at a time until directory empty>

	mv `head -100 output.txt` $dir2

else

	echo "no files to move.."

fi

I know I can accomplish what I need with a brute force creation of many files to read from but there must be a more sophisticated way to do it.

Is there way to drop the list of files into an array and do something like:

Code:

mv $files[1-100] $dir2
mv $files[101-200] $dir2

I need to find a usable number of files to move at a time (could be 100, 5000, 10000) while not knowing how many files could be in $dir1 at any given time.

Any guidance is greatly appreciated.

TIA,

Herb

Last edited by rbatte1; 08-22-2017 at 06:26 AM.. Reason: Added CODE tags

hburnswell

View Public Profile for hburnswell

Find all posts by hburnswell

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Move all files in a directory tree to a signal directory?

Is this possible? Let me know If I need specify further on what I am trying to do- I just want to spare you the boring details of my personal file management. Thanks in advance- Brian-

2. UNIX for Dummies Questions & Answers

How to move files between 2 dates from one directory to another

Hi All, I am coding for a requirement where I need to move files (filename.yymmdd) from one directory(A) to another(B) based on 2 date fields in a paramtere file. (Paramfile.txt) For e.g: In Paramfile.txt, BUS_DT =20120612 SUB_DT =20120602 In this case, i need to move all the files...

3. UNIX for Dummies Questions & Answers

Zip all files in a directory and move to another directory

Hi, need to zip all files in a directory and move to another directory after the zip.. i am using this one but didnt help me... zip -r my_proj_`date +%Y%m%d%H%MS`.zip /path/my_proj mv in_proj_`date +%Y%m%d%H%M%S`.zip /path/source/ i am trying to zip all the files in my_proj...

4. Shell Programming and Scripting

Process files in a directory and move them

I have a directory e2e_ms_xfer/cent01 this contains the multiple files some of which will be named below with unique date time stamps e2e_ms_edd_nom_CCYYMMDD_HHMM.csv What I want to do is in a loop 1) Get the oldest file 2) Rename 3) Move it up one level from e2e_ms_xfer/cent01 to...

5. UNIX for Dummies Questions & Answers

Rename files in a directory and move them

6. Shell Programming and Scripting

List files with date, create directory, move to the created directory

Hi all, i have a folder, with tons of files containing as following, on /my/folder/jobs/ some_name_2016-01-17-22-38-58_some name_0_0.zip.done some_name_2016-01-17-22-40-30_some name_0_0.zip.done some_name_2016-01-17-22-48-50_some name_0_0.zip.done and these can be lots of similar files,...

7. UNIX for Beginners Questions & Answers

Move all files one directory level up

I want to move all the files in a given directory up one level. For example: Dir1 Subdir1 I want to move all the files in Subdir1 up to Dir1 (then I want to ultimately delete Subdir1) Thanks, Ted

8. UNIX for Dummies Questions & Answers

How to move gz files from one source directory to destination directory?

Hi All, Daily i am doing the house keeping in one of my server and manually moving the files which were older than 90 days and moving to destination folder. using the find command . Could you please assist me how to put the automation using the shell script . ...

9. Shell Programming and Scripting

How Create new directory and move files to that directory.?

Hi All, We have main directory called "head" under this we have several sub directories and under these directories we have sub directories. My requirement is I have to find the SQL files which are having the string "procedure" under "head" directory and sub directories as well. And create...

LEARN ABOUT LINUX

alien::package::rpm

Alien::Package::Rpm(3pm)				User Contributed Perl Documentation				  Alien::Package::Rpm(3pm)

NAME

       Alien::Package::Rpm - an object that represents a rpm package

DESCRIPTION

       This is an object class that represents a rpm package. It is derived from Alien::Package.

FIELDS

       prefixes
	   Relocatable rpm packages have a prefixes field.

METHODS

       checkfile
	   Detect rpm files by their extention.

       install
	   Install a rpm. If RPMINSTALLOPT is set in the environement, the options in it are passed to rpm on its command line.

       scan
	   Implement the scan method to read a rpm file.

       unpack
	   Implement the unpack method to unpack a rpm file. This is a little nasty because it has to handle relocatable rpms and has to do a bit
	   of permissions fixing as well.

       prep
	   Prepare for package building by generating the spec file.

       cleantree
	   Delete the spec file.

       build
	   Build a rpm. If RPMBUILDOPT is set in the environement, the options in it are passed to rpm on its command line.

	   An optional parameter, if passed, can be used to specify the program to use to build the rpm. It defaults to rpmbuild.

       version
	   Set/get version.

	   When retreiving the version, remove any dashes in it.

       postinst
       postrm
       preinst
       prerm
	   Set/get script fields.

	   When retrieving a value, we have to do some truely sick mangling. Since debian/slackware scripts can be anything -- perl programs or
	   binary files -- and rpm is limited to only shell scripts, we need to encode the files and add a scrap of shell script to make it
	   unextract and run on the fly.

	   When setting a value, we do some mangling too. Rpm maintainer scripts are typically shell scripts, but often lack the leading shebang
	   line.  This can confuse dpkg, so add the shebang if it looks like there is no shebang magic already in place.

	   Additionally, it's not uncommon for rpm maintainer scripts to contain bashisms, which can be triggered when they are ran on systems
	   where /bin/sh is not bash. To work around this, the shebang line of the scripts is changed to use bash.

	   Also, if the rpm is relocatable, the script could refer to RPM_INSTALL_PREFIX, which is set by rpm at run time. Deal with this by
	   adding code to the script to set RPM_INSTALL_PREFIX.

       arch
	   Set/get arch field. When the arch field is set, some sanitizing is done first to convert it to the debian format used internally. When
	   it's retreived it's converted back to rpm form from the internal form.

AUTHOR

       Joey Hess <joey@kitenet.net>

perl v5.12.3							    2011-06-11						  Alien::Package::Rpm(3pm)