Copying Thousands of Tiny or Empty Files? Post: 302311383

Sponsored Content

Top Forums UNIX for Advanced & Expert Users Copying Thousands of Tiny or Empty Files? Post 302311383 by deckard on Tuesday 28th of April 2009 01:31:03 PM

04-28-2009

Registered User

Copying Thousands of Tiny or Empty Files?

There is a procedure I do here at work where I have to synchronize file systems. The source file system always has three or four directories of hundreds of thousands of tiny (1k or smaller) or empty files. Whenever my rsync command reaches these directories, I'm waiting for hours for those files to finish copying. Is there any way to decrease the time it takes for those files to be copied?

The files are generated by an application that definitely needs them, and I'm in no position to dispense with them. I wondered about trying to 'tar' the directories first, but I suspect that if I do, I'll merely be moving the time spent copying them during rsync to the time spent to create the archive in the first place.

My rsync command is pretty basic:

Code:

rsync -auvlxHS /source_dir/ /dest_dir/

Usually /dest_dir/ is a new, empty file system so it really is a full copy, but sometimes there are actual synchronizations done. However, if there's a better approach than my rsync, I'd like to know.

deckard

View Public Profile for deckard

Find all posts by deckard

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Finding a specific pattern from thousands of files ????

Hi All, I want to find a specific pattern from approximately 400000 files on solaris platform. Its very heavy for me to grep that pattern to each file individually. Can anybody suggest me some way to search for specific pattern (alpha numeric) from these forty thousand files. Please note that...

2. Shell Programming and Scripting

trnsmiting thousands ftp files and get an error message

Im transmiting thousands ftp files to a server, when type the command mput *, an error comes and say. args list to long. set to I. So ihave to transmit them in batch or blocks, but its too sloww. what shoul i do?. i need to do a program, or with a simple command i could solve the problem?

3. Shell Programming and Scripting

help to parallelize work on thousands of files

I need to find a smarter way to process about 60,000 files in a single directory. Every night a script runs on each file generating a output on another directory; this used to take 5 hours, but as the data grows it is taking 7 hours. The files are of different sizes, but there are 16 cores...

4. Shell Programming and Scripting

Search for patterns in thousands of files

Hi All, I want to search for a certain string in thousands of files and these files are distributed over different directories created daily. For that I created a small script in bash but while running it I am getting the below error: /ms.sh: xrealloc: subst.c:5173: cannot allocate...

5. Shell Programming and Scripting

Bash-awk to process thousands of files

Hi to all, I have thousand of files in a folder with names with format "FILE-YYYY-MM-DD-HHMM" for what I want to send the following AWK command awk '/Code.*/' FILE-2014* I'd like to separate all files that have the same date to a folder named with the corresponding date. For example, if I...

LEARN ABOUT DEBIAN

dtc-xen_migrate

dtc-xen_migrate(8)					      System Manager's Manual						dtc-xen_migrate(8)

NAME

       dtc-xen_migrate - migrate a VPS to another Xen server

SYNOPSIS

       LOCAL_VPS_ID DESTINATION_HOSTNAME [ DEST_VPS_ID ]

DESCRIPTION

       dtc-xen_migrate	This  shell  script  is a part of the dtc-xen package that is to be used by the dtc panel to manage a Xen VPS server. Note
       that you need rsync in both the source and the destination dom0.

       This script is used to move the content of a Xen virtual machine (VM) managed with dtc-xen to another Xen server of the same type.  The	VM
       destination will first be prepared, then the local VM will be shut down, then it's content will be transfered over using rsync.

       Note that this is NOT a live migration, and down time of approximatively the time needed for the rsync will occure.

PARAMETERS

       LOCAL_VPS_ID The DTC-Xen id of the VM, from 00 to 99.

       DESTINATION_HOSTNAME Destination Xen server hostname to migrate to.

       DEST_VPS_ID Destination DTC-Xen id of the VM, from 00 to 99.

EXAMPLE

       dtc-xen_migrate 19 alpha.example.com 21

       This will migrate the content of xen19 on the local machine to the destination server alpha.example.com using xen21 as VM name.

SEE ALSO

       dtc_reinstall_os(8), dtc_kill_vps_disk(8)

																dtc-xen_migrate(8)