Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Using UNIX Commands with Larger number of Files Post 302729585 by Don Cragun on Saturday 10th of November 2012 12:07:07 PM
Old 11-10-2012
Quote:
Originally Posted by RudiC
May I advise against executing a script within a find command? This would include, for every file found, creating a shell to run the script, implying a huge overhead, esp. in this case with "Large Number of Files". Why not collect all filenames in a working file and then work on that?
The whole point of my suggestion to use a script was to use the script to rearrange arguments to be passed to a simple command thereby allowing find's -exec primary to exec the underlying UNIX command with a large number of file operands instead of with a single file operand. I tried an experiment using the following commands on my relatively old MacBook Pro running Mac OS X Version 10.7.5, which sets {ARG_MAX} to 262144 bytes (i.e., 256Kb). I used cp as the test command to copy all of the PDF files found in and under my home directory to /tmp/pdfdest. /tmp/pdfdest was a new directory when I started this test, but I did not empty and recreate the directory between tests.

I used the command:
Code:
time find $HOME -name '*.pdf' -exec cp -f {} /tmp/pdfdest \;

three times and ignored the 1st time (which ran a lot slower than the other two as it cleared out the various caches and loaded my home directory's file hierarchy). The two remaining runs averaged 2 minutes 27.45 seconds wall clock time, 0.85 seconds user clock time, and 5.56 seconds system time to copy 881 files by invoking cp 881 times.

Adding the following Korn shell script (named CpDest1st):
Code:
#!/bin/ksh
# Usage: CpDest1st options destdir srcfile...
opts="$1"
dest="$2"
shift 2
exec cp $opts "$@" "$dest"

and using the command:
Code:
time find $HOME -name '*.pdf' -exec CpDest1st -f /tmp/pdfdest {} +

three times and again ignored the first set of results and averaged the other two sets of results. Copying the same 881 files using this method invoking the shell script and the cp utility once each took 2 minutes 3.02 seconds wall time, 0.42 seconds user time, and 3.76 seconds system time.

This isn't a statistically valid comparison and your mileage will vary depending on the value of {ARG_MAX} on your system, the number and sizes of the PDF files being copied, ... . But, it does show that the overhead of using a shell script to reduce the number of invocations of cp (or mv or many other utilities) may actually reduce the elapsed time and the amount of system resources used. In this simple test, using the shell script reduced wall clock time 16%, reduced system time 50%, and reduced user time 32%.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

sending larger files via ftp

hi all, i am looking for ways to make ftp efficient by tuning the parameters currently, tcp_max_buf is 1 MB tcp_xmit_hiwat is 48 KB say to transmit multiple 2 gb files from unix server to mainframe sys, will increasing the window size or the send buffer size of the current TCP/IP... (6 Replies)
Discussion started by: matrixmadhan
6 Replies

2. Shell Programming and Scripting

How to initialize array with a larger number?

Language: ksh OS: SunOS I have been getting the 'subscript out of range' error when the below array variable gets elements greater that 1024. I understand that 1024 is the default size for 'set -A' dynamic array, but is there a way to initialize it with a larger number? set -A arr `grep... (6 Replies)
Discussion started by: ChicagoBlues
6 Replies

3. AIX

Tar files larger than 2GB

Hi, Does anyone know if it is possible to tar files larger than 2GB? The reason being is they want me to dump a single file (which is around 20GB) to a tape drive and they will restore it on a Solaris box. I know the tar have a limitation of 2GB so I am thinking of a way how to overcome this.... (11 Replies)
Discussion started by: depam
11 Replies

4. UNIX for Dummies Questions & Answers

unix commands related to ftp of files..

Hi, I am new to unix , I was planning to write a script that will FTP files to destination folder. , Please guide me what are the various networking commands that unix will help in this ftp process..?:confused: (1 Reply)
Discussion started by: rahul125
1 Replies

5. Programming

Using basic UNIX commands to make/compile JAVA files

Hello! This is my first post, and I just learned what UNIX was this week. For a JAVA programming class I am taking, I must be able to create a directory in UNIX, use the nano command to create a JAVA program, compile it, and then run it on the command prompt using the java command. For some... (5 Replies)
Discussion started by: UNdvoItX
5 Replies

6. UNIX for Dummies Questions & Answers

7z command for files larger than 4GB ( unzip doesn't work)

My unzip command doesn't work for files that are greater than 4GB. Consider my file name is unzip -p -a filename.zip, the command doesn't work since the size of the file is larger. I need to know the corresponding 7z command for the same. This is my Unix shell script program: if then ... (14 Replies)
Discussion started by: chandraprakash
14 Replies

7. Shell Programming and Scripting

Backingup larger files with TAR command

I need to backup my database but the files are very large and the TAR command will not let me. I searched aids and found that I could do something with the mknod, COMPRESS and TAR command using them together. I appreciate your help. (10 Replies)
Discussion started by: frizcala
10 Replies

8. UNIX for Dummies Questions & Answers

Split larger files into smaller ones with Column names

Hi, I have one large files of 100000 rows with header column. Eg: Emp Code, Emp Name 101,xxx 102,YYY 103,zzz ... ... I want to split the files into smaller files with only 30000 rows each..File 1,2 and 3 must have 30000 rows and file 4 must contain 10000 rows. But the column... (1 Reply)
Discussion started by: Nivas
1 Replies

9. UNIX for Beginners Questions & Answers

Need to select files larger than 500Mb from servers

I need help modifying these two scripts to do the following: - print files in (MB) instead of (KB) - only select files larger than 500MB -> these will be mailed out daily - Select all files regardless of size all in (MB) -> these will be mailed out once a week this is what i have so far and... (5 Replies)
Discussion started by: donpasscal
5 Replies

10. UNIX for Beginners Questions & Answers

Help with Expect script for pulling log files size larger than 500Mb;

I am new at developing EXPECT scripts. I'm trying to create a script that will automatically connect to a several UNIX (sun solaris and HPUX) database server via FTP and pull the sizes of the listener/alert log files from specified server directory on the remote machines. 1. I want the script... (7 Replies)
Discussion started by: mikebantor
7 Replies
LOCALEDEF(1)						      General Commands Manual						      LOCALEDEF(1)

NAME
localedef - compile locale definition files SYNOPSIS
localedef [options] outputpath localedef --list-archive [options] localedef --delete-from-archive [options] localename ... localedef --add-to-archive [options] compiledpath localedef --version localedef --help localedef --usage DESCRIPTION
The localedef program reads the indicated charmap and input files, compiles them to a binary form quickly usable by the locale(7) functions in the C library, and places the output in outputpath. If outputpath contains a slash character ('/'), it is directly the name of the output directory. In this case, there is a separate output file for each locale category (LC_CTIME, LC_NUMERIC, and so on). Otherwise, if the --no-archive option is used, outputpath is the name of a subdirectory in /usr/lib/locale where per-category compiled files are placed. Otherwise, outputpath is the name of a locale and the compiled locale data is added to the archive file /usr/lib/locale/locale-archive. In any case, localedef aborts if the directory in which it tries to write locale files has not already been created. If no charmapfile is given, the value POSIX is used by default. If no inputfile is given, or if it is given as a dash (-), localedef reads from standard input. OPTIONS
Most options can have either short or long forms. If multiple short options are used, they can be combined in one word (for example, -cv is identical to -c -v). If a short option takes an argument, the argument can be given separately as the next word (-f foo), or it can be written together with the option letter (-ffoo). If a long option takes an argument, the argument can be given separately as the next word, or it can be written as option=argument (--charmap=foo). Operation selection options A few options direct localedef to do something else than compile locale definitions. Only one of these should be used at a time. --delete-from-archive Delete the named locales from the locale archive file. --list-archive List the locales contained in the locale archive file. --add-to-archive Add the compiledpath directories to the locale archive file. The directories should have been created by previous runs of lo- caledef, using --no-archive. Other options Some of the following options are only sensible for some operations; hopefully it is self-evident which ones. -f charmapfile, --charmap=charmapfile Specify the file that defines the symbolic character names that are used by the input file. If the file is in the default directory for character maps, it is not necessary to specify the full pathname. This default directory is printed by localedef --help. -i inputfile, --inputfile=inputfile Specify the locale definition file to compile. If inputfile is not absolute, localedef will also look in the directories specified by the environment variable I18NPATH and in the default directory for locale definition files. This default directory is printed by localedef --help. -u repertoirefile, --repertoire-map=repertoirefile Read mappings from symbolic names to Unicode UCS4 values from repertoirefile. -A aliasfile, --alias-file=aliasfile Use aliasfile to look up aliases for locale names. There is no default aliases file. --prefix=pathname Set prefix to be prepended to the full archive pathname. By default, the prefix is empty. Setting the prefix to foo, the archive would be placed in foo/usr/lib/locale/locale-archive. -c, --force Write the output files even if warnings were generated about the input file. --old-style Create old-style tables. -v, --verbose Generate extra warnings about errors that are normally ignored. --quiet Suppress all notifications and warnings, and report only fatal errors. --posix Conform strictly to POSIX. Implies --verbose. This option currently has no other effect. Posix conformance is assumed if the en- vironment variable POSIXLY_CORRECT is set. --replace Replace a locale in the locale archive file. Without this option, if the locale is in the archive file already, an error occurs. --no-archive Do not use the locale archive file, instead create outputpath as a subdirectory in the same directory as the locale archive file, and create separate output files for locale categories in it. --help Print a usage summary and exit. Also prints the default paths used by localedef. --usage Print a short usage summary and exit. -V, --version Print the version number, license, and disclaimer of warranty for localedef. ENVIRONMENT
POSIXLY_CORRECT The --posix flag is assumed if this environment variable is set. I18NPATH A colon separated list of default directories for locale definition files. FILES
/usr/share/i18n/charmaps Usual default charmap path. /usr/share/i18n/locales Usual default path for locale source files. /usr/share/i18n/repertoiremaps Usual default repertoire map path. /usr/lib/locale/locale-archive Usual default locale archive location. outputpath/LC_COLLATE One of the output files. It describes the rules for comparing strings in the locale's alphabet. outputpath/LC_CTYPE One of the output files. It contains information about character cases and case conversions for the locale. outputpath/LC_MONETARY One of the output files. It describes the way monetary values should be formatted in the locale. outputpath/LC_MESSAGES/SYS_LC_MESSAGES One of the output files. It contains information about the language messages should be printed in, and what an affirmative or nega- tive answer looks like. outputpath/LC_NUMERIC One of the output files. It describes the rules for formatting numbers in the locale. outputpath/LC_TIME One of the output files. It describes the rules for formatting times and dates in the locale. outputpath/LC_PAPER One of the output files. It describes the default paper size in the locale. outputpath/LC_NAME One of the output files. It describes the rules for formatting names in the locale. outputpath/LC_ADDRESS One of the output files. It describes the rules for formatting addresses, and other location information in the locale. outputpath/LC_TELEPHONE One of the output files. It describes the rules for formatting telephone numbers in the locale. outputpath/LC_MEASUREMENT One of the output files. It describes the rules for measurement in the locale, e.g. Metric or other units. outputpath/LC_IDENTIFICATION One of the output files. It identifies the elements within the locale. EXAMPLES
Compile the locale files for Finnish in the UTF-8 character set and add it to the default locale archive with the name fi_FI.UTF-8: localedef -f UTF-8 -i fi_FI fi_FI.UTF-8 The same, but generate files into the current directory (note that the last argument must then contain a slash): localedef -f UTF-8 -i fi_FI ./ SEE ALSO
locale(5), locale(7), locale(1) AUTHOR
The program was written by Ulrich Drepper. This manual page was written by Richard Braakman <dark@xs4all.nl> on behalf of the Debian GNU/Linux Project and anyone else who wants it. It was amended by Alastair McKinstry <mckinstry@computer.org> to explain new ISO 14652 elements, and amended further by Lars Wirzenius <liw@iki.fi> to document new functionality (as of GNU C library 2.3.5). The manpage is not supported by the GNU libc maintainers and may be out of date. STANDARDS
This program conforms to the POSIX standard P1003.2 May 20, 2005 LOCALEDEF(1)
All times are GMT -4. The time now is 10:23 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy