UNIX for Advanced & Expert Users

Hi guys,

I have an issue processing a large amount of files. I have around 5 million files (some of them are actually directories) in a server.

I am unable to find out the exact number of files since it's taking "forever" to finish (See this thread for more on the issue).

Anyway, now I want to move these ~5 million files to a different location so my first thought was to tar/gzip the files and SCP them somewhere else as a single file however the tar process is also taking a loooong time to finish (in fact it never finished and I cancelled the job after 10 hours).

Basically I just want to build a single package containing the ~5 million files (zip, tar, cpio, raw data, whatever) so that I can easily move and transfer the files to a different location.

Any ideas?

Thank you.

Here is your problem - you are reading millions of directory entries and writing to a tarfile.
Then you copy the tarfile somewhere, then extract. Tons of I/O writing the tarball, I/O copying it, I/O extracting it.

Eliminate the "middleman I/O".

FWIW:
Plus, assuming you actually want the data, you are perpetuating the problem - way too many file entries per directory. You really should reorganize the directory structure. It is probably not possible that users are reading those files very often or you would have lots of user complaints 'It takes forever to read a file...'

That said:
moving a directory tree from node to node, eliminate the middleman processing:

Code:

scp -r -p /my/path remotenode:/my/path

using tar to relocate on the same box, eliminate the middleman

Code:

tar cf - /my/path | ( cd /new/target; tar xfp -)

BTW don't kill this job off until it is done - it could take forever. You will never make any progress if you kill of these jobs.

When you say somewhere else, I guess you mean a different server ?
I imagine the "gzip" and the ssl encryption in scp are both adding quite an overhead.

There is a tradeoff between network speed, cpu speed and disk speed. You may find it more efficient leaving out the gzip if you have a fast network. Also if you are not worried about security using "rsh" instead of "ssh/scp" will be quicker. Probably the quickest would be:

Code:

tar cf - my/path | rsh -l user host "cd /new/target; tar xfp -"

However if network speed is slower and you need security something like "rsync" with the "-z" option may be better for you.

Please state what Operating System you have and describe your hardware configuration including memory and discs and enything relevant to performance.

Is it safe to assume that the filesystem will be quiescent?

Do you have spare discs equivalent to say twice the existing space? Personally I would copy the entire filesystem first to produce a defragmented filesystem which runs at a reasonable speed. This is also intended to prove that the original disc can be read from end-to-end.

Please post the current values of:

Code:

df -i
df -k

First of all, thanks all for your suggestions...

Well, even though I have a lot of files, each one of them is pretty small.

I'd say that the whole ~5 million text files do not take more than 50 GB of disk space*

Some details about my system:

Code:

RHEL 5.5 (Tikanga)
Kernel 2.6.18 x86_64
Local ext3 LVM (3 Physical Volumes)
4.5 GB RAM

My Disks:

Code:

SCSI device sdb: 31457280 512-byte hdwr sectors
sd 0:0:0:0: Attached scsi disk sdb
  Type:   Direct-Access                      ANSI SCSI revision: 02
 target0:0:1: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)

Code:

[root@atlas ~]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/volAvg-A1lv
                     15466496 4464503 11001993   29% /export
					 
[root@atlas ~]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/volAvg-A1lv
                     121790564  89066984  26538244  78% /export

*According to df -k I'm using around 80 GB but that's because I have other files in the same filesystem that are eating up close to 30 GB.

A different server or a different filesystem within the same server; whatever approach is faster.

Not exactly quiescent but with very little disk usage since I will run this process at night when nobody uses the server.

Yes, I can attach more disks if necessary.

Different filesystems on the same server will be orders of magnitude quicker.
Probably one of the fastest is the one mentioned previously:

Code:

tar cf - /my/path | ( cd /new/target; tar xfp -)

And this has the added benefit of making the files contiguous on the new filesystem.
This also doesnt compress or encrypt the files, both of which will hit the cpu.

Guys,

Just wanted to thank you.

Since the original request was creating more issues than expected I have opted for a whole disk backup (the disk is not that big)... Faster and less problematic.

Thanks for your valuable suggestions, though.