Copying Thousands of Tiny or Empty Files? Post: 302311383

Sponsored Content

Top Forums UNIX for Advanced & Expert Users Copying Thousands of Tiny or Empty Files? Post 302311383 by deckard on Tuesday 28th of April 2009 01:31:03 PM

04-28-2009

Registered User

Copying Thousands of Tiny or Empty Files?

There is a procedure I do here at work where I have to synchronize file systems. The source file system always has three or four directories of hundreds of thousands of tiny (1k or smaller) or empty files. Whenever my rsync command reaches these directories, I'm waiting for hours for those files to finish copying. Is there any way to decrease the time it takes for those files to be copied?

The files are generated by an application that definitely needs them, and I'm in no position to dispense with them. I wondered about trying to 'tar' the directories first, but I suspect that if I do, I'll merely be moving the time spent copying them during rsync to the time spent to create the archive in the first place.

My rsync command is pretty basic:

Code:

rsync -auvlxHS /source_dir/ /dest_dir/

Usually /dest_dir/ is a new, empty file system so it really is a full copy, but sometimes there are actual synchronizations done. However, if there's a better approach than my rsync, I'd like to know.

deckard

View Public Profile for deckard

Find all posts by deckard

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Finding a specific pattern from thousands of files ????

Hi All, I want to find a specific pattern from approximately 400000 files on solaris platform. Its very heavy for me to grep that pattern to each file individually. Can anybody suggest me some way to search for specific pattern (alpha numeric) from these forty thousand files. Please note that...

2. Shell Programming and Scripting

trnsmiting thousands ftp files and get an error message

Im transmiting thousands ftp files to a server, when type the command mput *, an error comes and say. args list to long. set to I. So ihave to transmit them in batch or blocks, but its too sloww. what shoul i do?. i need to do a program, or with a simple command i could solve the problem?

3. Shell Programming and Scripting

help to parallelize work on thousands of files

I need to find a smarter way to process about 60,000 files in a single directory. Every night a script runs on each file generating a output on another directory; this used to take 5 hours, but as the data grows it is taking 7 hours. The files are of different sizes, but there are 16 cores...

4. Shell Programming and Scripting

Search for patterns in thousands of files

Hi All, I want to search for a certain string in thousands of files and these files are distributed over different directories created daily. For that I created a small script in bash but while running it I am getting the below error: /ms.sh: xrealloc: subst.c:5173: cannot allocate...

5. Shell Programming and Scripting

Bash-awk to process thousands of files

Hi to all, I have thousand of files in a folder with names with format "FILE-YYYY-MM-DD-HHMM" for what I want to send the following AWK command awk '/Code.*/' FILE-2014* I'd like to separate all files that have the same date to a folder named with the corresponding date. For example, if I...

LEARN ABOUT DEBIAN

tap::parser::yamlish::reader

TAP::Parser::YAMLish::Reader(3pm)			User Contributed Perl Documentation			 TAP::Parser::YAMLish::Reader(3pm)

NAME

       TAP::Parser::YAMLish::Reader - Read YAMLish data from iterator

VERSION

       Version 3.25

SYNOPSIS

DESCRIPTION

       Note that parts of this code were derived from YAML::Tiny with the permission of Adam Kennedy.

METHODS

   Class Methods
       "new"

       The constructor "new" creates and returns an empty "TAP::Parser::YAMLish::Reader" object.

	my $reader = TAP::Parser::YAMLish::Reader->new;

   Instance Methods
       "read"

	my $got = $reader->read($iterator);

       Read YAMLish from a TAP::Parser::Iterator and return the data structure it represents.

       "get_raw"

	my $source = $reader->get_source;

       Return the raw YAMLish source from the most recent "read".

AUTHOR

       Andy Armstrong, <andy@hexten.net>

       Adam Kennedy wrote YAML::Tiny which provided the template and many of the YAML matching regular expressions for this module.

SEE ALSO

       YAML::Tiny, YAML, YAML::Syck, Config::Tiny, CSS::Tiny, <http://use.perl.org/~Alias/journal/29427>

COPYRIGHT

       Copyright 2007-2011 Andy Armstrong.

       Portions copyright 2006-2008 Adam Kennedy.

       This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

       The full text of the license can be found in the LICENSE file included with this module.

perl v5.14.2							    2012-06-05					 TAP::Parser::YAMLish::Reader(3pm)