Shell Programming and Scripting

I am in a situation where I have some hundreds of thousands of files in a directory (name of directory: big_directory). Now, working on this directory is extremely slow.

I now plan to do this:

Copy 1000 files in each of the directories by creating the directories automatically. All files have .dat extension in the directory and have names like 1.dat, 2.dat, 3.dat until the last file number

This is what I intend to do:

1. Automatically create a directory "dir1" and copy first 1000 ".dat" files (present in big_directory) in it.

2. Automatically create a directory "dir2" and then copy the next 1000 ".dat" files (present in big_directory) in it.

3. Automatically create a directory "dir3" and then copy the next 1000 ".dat" files (present in big_directory) in it.

4. and so on until last file

I am using Linux with BASH. I know how to copy first 1000 files but not sure how to create directories automatically using a counter and copy them there.

Code:

cp `ls -ltr | grep ^- | head -1000 | tr -s " " | cut -d " " -f9` destination

This should do:

Code:

#!/bin/bash

count=`find -name "*.dat" | wc -l`
iterations=$(($count/1000))

for j in $(seq $iterations); do
        dir="dir$j"
        mkdir $dir
        mv $j???.dat $dir
done

I used mv instead of cp... Your call, but mv will be faster and with cp You will double this hundreds of thousands of files...

PS.
Oh, and add following line at the begining, before the loop, to copy the first 1000 files:

Code:

mkdir dir0 && mv ?.dat ??.dat ???.dat dir0/