Copying Thousands of Tiny or Empty Files? Post: 302311391

Sponsored Content

Top Forums UNIX for Advanced & Expert Users Copying Thousands of Tiny or Empty Files? Post 302311391 by jim mcnamara on Tuesday 28th of April 2009 01:48:52 PM

04-28-2009

Registered User

Can you run multiple 'threads' of rsync - divide up the source tree and dest tree among several rsync processes?

Code:

rsync -auvlxHS /source_dir/dir1 /dest_dir/dir1
rsync -auvlxHS /source_dir/dir2 /dest_dir/dir2
rsync -auvlxHS /source_dir/dir3 /dest_dir/dir3

When you create lots of files and directories there is substantially more filesystem overhead than just writing to an existing file. You may want to do some serious filesystem tuning on the destination box, particularly the /dest_dir filesystem.

Also, having huge numbers of files in a single directory really bogs things down as well. readdir() takes a lot longer to complete a full scan of a directory for example...

What OS?

jim mcnamara

View Public Profile for jim mcnamara

Find all posts by jim mcnamara

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Finding a specific pattern from thousands of files ????

Hi All, I want to find a specific pattern from approximately 400000 files on solaris platform. Its very heavy for me to grep that pattern to each file individually. Can anybody suggest me some way to search for specific pattern (alpha numeric) from these forty thousand files. Please note that...

2. Shell Programming and Scripting

trnsmiting thousands ftp files and get an error message

Im transmiting thousands ftp files to a server, when type the command mput *, an error comes and say. args list to long. set to I. So ihave to transmit them in batch or blocks, but its too sloww. what shoul i do?. i need to do a program, or with a simple command i could solve the problem?

3. Shell Programming and Scripting

help to parallelize work on thousands of files

I need to find a smarter way to process about 60,000 files in a single directory. Every night a script runs on each file generating a output on another directory; this used to take 5 hours, but as the data grows it is taking 7 hours. The files are of different sizes, but there are 16 cores...

4. Shell Programming and Scripting

Search for patterns in thousands of files

Hi All, I want to search for a certain string in thousands of files and these files are distributed over different directories created daily. For that I created a small script in bash but while running it I am getting the below error: /ms.sh: xrealloc: subst.c:5173: cannot allocate...

5. Shell Programming and Scripting

Bash-awk to process thousands of files

Hi to all, I have thousand of files in a folder with names with format "FILE-YYYY-MM-DD-HHMM" for what I want to send the following AWK command awk '/Code.*/' FILE-2014* I'd like to separate all files that have the same date to a folder named with the corresponding date. For example, if I...

LEARN ABOUT DEBIAN

e2cp

E2CP(1) 						      General Commands Manual							   E2CP(1)

NAME

       e2cp - copies files from/to an e2fs filesystem

SYNOPSIS

       e2cp [options] files... dest

DESCRIPTION

       The e2cp command copies files to/from an e2fs filesystem from/to the machine's filesystem.

OPTIONS

       A single `-' (dash) can be used instead of a file name to indicate standard input/output.

       -0     Input lines terminated by a null character.

       -a     Absolute	directory  names  - create directories instead of just copying into the destination. Only valid for copying into an ext2fs
	      filesystem.

       -d dest_dir
	      Destination of files to be copied. May be in the ext2fs filesystem or the host filesystem.

       -G gid Set the default group to gid.

       -O uid Set the default file owner to uid.

       -p     Preserve host file attributes (permissions, times, etc.) when copying files.

       -P mode
	      Set the file permissions (ie 755, 666).

       -s src_dir
	      The source of the files to be copied.

       -v     Be verbose.

SEE ALSO

       e2tools(7), e2ln(1), e2ls(1), e2mkdir(1), e2mv(1), e2rm(1), e2tail(1).

AUTHOR

       The e2tools were written by Keith Sheffield <sheff@pobox.com>.

       This manual page was written by Lucas Wall <lwall@debian.org>, for the Debian project (but may be used by others).

								   March 2, 2005							   E2CP(1)