Sponsored Content
Top Forums Shell Programming and Scripting Move files from one directory to another in chunks Post 303002302 by hburnswell on Monday 21st of August 2017 11:46:49 PM
Old 08-22-2017
Move files from one directory to another in chunks

All,

I have an application that is not working properly and the company is 'in the process' of fixing it. In the meantime, I want to write a bash script work-around. However, what I thought was going to be simple is seemingly not.

Need:

- Move files from one directory to another in chunks.

If I move all the files at once the system resources are heavily consumed and the application suffers. The source directory ($dir1) will contain any number of files (100's or 1000's) at any given time. So, I'd like to do something like:

Code:
if [ "$(ls -A $dir1) ]; then

	find $dir1 -maxdepth 1 -type f > output.txt

	<loop through output.txt moving x amount of files at a time until directory empty>

	mv `head -100 output.txt` $dir2

else

	echo "no files to move.."

fi

I know I can accomplish what I need with a brute force creation of many files to read from but there must be a more sophisticated way to do it.

Is there way to drop the list of files into an array and do something like:

Code:
mv $files[1-100] $dir2
mv $files[101-200] $dir2

I need to find a usable number of files to move at a time (could be 100, 5000, 10000) while not knowing how many files could be in $dir1 at any given time.

Any guidance is greatly appreciated.

TIA,

Herb

Last edited by rbatte1; 08-22-2017 at 06:26 AM.. Reason: Added CODE tags
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Move all files in a directory tree to a signal directory?

Is this possible? Let me know If I need specify further on what I am trying to do- I just want to spare you the boring details of my personal file management. Thanks in advance- Brian- (2 Replies)
Discussion started by: briandanielz
2 Replies

2. UNIX for Dummies Questions & Answers

How to move files between 2 dates from one directory to another

Hi All, I am coding for a requirement where I need to move files (filename.yymmdd) from one directory(A) to another(B) based on 2 date fields in a paramtere file. (Paramfile.txt) For e.g: In Paramfile.txt, BUS_DT =20120612 SUB_DT =20120602 In this case, i need to move all the files... (14 Replies)
Discussion started by: dsfreddie
14 Replies

3. UNIX for Dummies Questions & Answers

Zip all files in a directory and move to another directory

Hi, need to zip all files in a directory and move to another directory after the zip.. i am using this one but didnt help me... zip -r my_proj_`date +%Y%m%d%H%MS`.zip /path/my_proj mv in_proj_`date +%Y%m%d%H%M%S`.zip /path/source/ i am trying to zip all the files in my_proj... (0 Replies)
Discussion started by: dssyadav
0 Replies

4. Shell Programming and Scripting

Process files in a directory and move them

I have a directory e2e_ms_xfer/cent01 this contains the multiple files some of which will be named below with unique date time stamps e2e_ms_edd_nom_CCYYMMDD_HHMM.csv What I want to do is in a loop 1) Get the oldest file 2) Rename 3) Move it up one level from e2e_ms_xfer/cent01 to... (1 Reply)
Discussion started by: andymay
1 Replies

5. UNIX for Dummies Questions & Answers

Rename files in a directory and move them

I have a directory e2e_ms_xfer/cent01 this contains the multiple files some of which will be named below with unique date time stamps e2e_ms_edd_nom_CCYYMMDD_HHMM.csv What I want to do is in a loop 1) Get the oldest file 2) Rename 3) Move it up one level from e2e_ms_xfer/cent01 to... (1 Reply)
Discussion started by: andymay
1 Replies

6. Shell Programming and Scripting

List files with date, create directory, move to the created directory

Hi all, i have a folder, with tons of files containing as following, on /my/folder/jobs/ some_name_2016-01-17-22-38-58_some name_0_0.zip.done some_name_2016-01-17-22-40-30_some name_0_0.zip.done some_name_2016-01-17-22-48-50_some name_0_0.zip.done and these can be lots of similar files,... (6 Replies)
Discussion started by: charli1
6 Replies

7. UNIX for Beginners Questions & Answers

Move all files one directory level up

I want to move all the files in a given directory up one level. For example: Dir1 Subdir1 I want to move all the files in Subdir1 up to Dir1 (then I want to ultimately delete Subdir1) Thanks, Ted (10 Replies)
Discussion started by: ftrobaugh
10 Replies

8. UNIX for Dummies Questions & Answers

How to move gz files from one source directory to destination directory?

Hi All, Daily i am doing the house keeping in one of my server and manually moving the files which were older than 90 days and moving to destination folder. using the find command . Could you please assist me how to put the automation using the shell script . ... (11 Replies)
Discussion started by: venkat918
11 Replies

9. Shell Programming and Scripting

How Create new directory and move files to that directory.?

Hi All, We have main directory called "head" under this we have several sub directories and under these directories we have sub directories. My requirement is I have to find the SQL files which are having the string "procedure" under "head" directory and sub directories as well. And create... (14 Replies)
Discussion started by: ROCK_PLSQL
14 Replies
scandir(3)						     Library Functions Manual							scandir(3)

NAME
scandir, alphasort - Scans or sorts directory contents LIBRARY
Standard C Library (libc.a) SYNOPSIS
#include <sys/types.h> #include <sys/dir.h> int scandir ( char *dir_name, struct dirent * (*name_list[ ]), int (*select) ( struct dirent *dir ), int (*compare) ( struct dirent **dir1, struct dirent **dir2 ) ); int alphasort ( struct dirent **dir1, struct dirent **dir2 ); PARAMETERS
Points to the directory name. Points to the array of pointers to directory entries. Points to a user-supplied function that is called by the scandir() function to select which entries to include in the array. Points to a user-supplied function that sorts the completed array. Points to a dirent structure. Points to a dirent structure. DESCRIPTION
The scandir() function reads the directory pointed to by the dir_name parameter. It then uses the malloc() function to create an array of pointers to directory entries. The scandir() function returns the number of entries in the array and, through the name_list parameter, a pointer to the array. The select parameter points to a user-supplied function that the scandir() function calls to select which entries to include in the array. The selection routine is passed a pointer to a directory entry and returns a nonzero value for a directory entry that is included in the array. If the select parameter is a null value, all directory entries are included. The compare parameter points to a user-supplied function that is passed to the qsort() function to sort the completed array. If the compare parameter is a null value, the array is not sorted. The memory allocated to the array can be deallocated by freeing each pointer in the array, and the array itself, with the free() function. The alphasort() function alphabetically compares the two dirent structures pointed to by the dir1 and dir2 parameters. This function can be passed as the compare parameter to either the scandir() function or the qsort() function. A user-supplied subroutine may also be used. RETURN VALUES
The scandir() function returns -1 if the directory cannot be opened for reading or if the malloc() function cannot allocate enough memory to hold all the data structures. If successful, the scandir() function returns the number of entries found. The alphasort() function returns the following values: Less than 0 (zero): The dirent structure pointed to by the dir1 parameter is lexi- cally less than the dirent structure pointed to by the dir2 parameter. 0 (zero): The dirent structures pointed to by the dir1 parameter and the dir2 parameter are equal. Greater than 0 (zero): The dirent structure pointed to by the dir1 parameter is lexically greater than the dirent structure pointed to by the dir2 parameter. RELATED INFORMATION
Functions: malloc(3), opendir(3), qsort(3) delim off scandir(3)
All times are GMT -4. The time now is 03:39 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy