Sponsored Content
Top Forums UNIX for Advanced & Expert Users Copying Thousands of Tiny or Empty Files? Post 302311391 by jim mcnamara on Tuesday 28th of April 2009 01:48:52 PM
Old 04-28-2009
Can you run multiple 'threads' of rsync - divide up the source tree and dest tree among several rsync processes?

Code:
rsync -auvlxHS /source_dir/dir1 /dest_dir/dir1
rsync -auvlxHS /source_dir/dir2 /dest_dir/dir2
rsync -auvlxHS /source_dir/dir3 /dest_dir/dir3

When you create lots of files and directories there is substantially more filesystem overhead than just writing to an existing file. You may want to do some serious filesystem tuning on the destination box, particularly the /dest_dir filesystem.

Also, having huge numbers of files in a single directory really bogs things down as well. readdir() takes a lot longer to complete a full scan of a directory for example...

What OS?
 

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Finding a specific pattern from thousands of files ????

Hi All, I want to find a specific pattern from approximately 400000 files on solaris platform. Its very heavy for me to grep that pattern to each file individually. Can anybody suggest me some way to search for specific pattern (alpha numeric) from these forty thousand files. Please note that... (6 Replies)
Discussion started by: aarora_98
6 Replies

2. Shell Programming and Scripting

trnsmiting thousands ftp files and get an error message

Im transmiting thousands ftp files to a server, when type the command mput *, an error comes and say. args list to long. set to I. So ihave to transmit them in batch or blocks, but its too sloww. what shoul i do?. i need to do a program, or with a simple command i could solve the problem? (3 Replies)
Discussion started by: alexcol
3 Replies

3. Shell Programming and Scripting

help to parallelize work on thousands of files

I need to find a smarter way to process about 60,000 files in a single directory. Every night a script runs on each file generating a output on another directory; this used to take 5 hours, but as the data grows it is taking 7 hours. The files are of different sizes, but there are 16 cores... (10 Replies)
Discussion started by: vhope07
10 Replies

4. Shell Programming and Scripting

Search for patterns in thousands of files

Hi All, I want to search for a certain string in thousands of files and these files are distributed over different directories created daily. For that I created a small script in bash but while running it I am getting the below error: /ms.sh: xrealloc: subst.c:5173: cannot allocate... (17 Replies)
Discussion started by: danish0909
17 Replies

5. Shell Programming and Scripting

Bash-awk to process thousands of files

Hi to all, I have thousand of files in a folder with names with format "FILE-YYYY-MM-DD-HHMM" for what I want to send the following AWK command awk '/Code.*/' FILE-2014* I'd like to separate all files that have the same date to a folder named with the corresponding date. For example, if I... (7 Replies)
Discussion started by: Ophiuchus
7 Replies
Module::Build::Tiny(3pm)				User Contributed Perl Documentation				  Module::Build::Tiny(3pm)

NAME
Module::Build::Tiny - A tiny replacement for Module::Build VERSION
version 0.034 SYNOPSIS
use Module::Build::Tiny; Build_PL(); DESCRIPTION
Many Perl distributions use a Build.PL file instead of a Makefile.PL file to drive distribution configuration, build, test and installation. Traditionally, Build.PL uses Module::Build as the underlying build system. This module provides a simple, lightweight, drop-in replacement. Whereas Module::Build has over 6,700 lines of code; this module has less than 120, yet supports the features needed by most distributions. Supported o Pure Perl distributions o Building XS or C o Recursive test files o MYMETA o Man page generation o Generated code from PL files Not Supported o Dynamic prerequisites o HTML documentation generation o Extending Module::Build::Tiny o Module sharedirs Directory structure Your .pm and .pod files must be in lib/. Any executables must be in script/. Test files must be in t/. Dist sharedirs must be in share/. USAGE
These all work pretty much like their Module::Build equivalents. perl Build.PL Build [ build ] Build test Build install This supports the following options: o verbose o install_base o installdirs o prefix o install_path o destdir o uninst o config o pure-perl o create_packlist AUTHORING
This module doesn't support authoring. To develop modules using Module::Build::Tiny, usage of Dist::Zilla::Plugin::ModuleBuildTiny or App::ModuleBuildTiny is recommended. CONFIG FILE AND ENVIRONMENT
Options can be provided in the "PERL_MB_OPT" environment variable the same way they can with Module::Build. This should be done during the configuration stage. Incompatibilities o Argument parsing Module::Build has an extremely permissive way of argument handling, Module::Build::Tiny only supports a (sane) subset of that. In particular, "./Build destdir=/foo" does not work, you will need to pass it as "./Build --destdir=/foo". o .modulebuildrc Module::Build::Tiny does not support .modulebuildrc files. In particular, this means that versions of local::lib older than 1.006008 may break. Upgrading it resolves this issue. SEE ALSO
Module::Build AUTHORS
o Leon Timmermans <leont@cpan.org> o David Golden <dagolden@cpan.org> COPYRIGHT AND LICENSE
This software is copyright (c) 2011 by Leon Timmermans, David Golden. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. perl v5.18.2 2014-01-21 Module::Build::Tiny(3pm)
All times are GMT -4. The time now is 06:10 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy