Sponsored Content
Top Forums Shell Programming and Scripting Fine Tune - Huge files/directory - Purging Post 302529857 by mirni on Friday 10th of June 2011 05:01:47 PM
Old 06-10-2011
Quote:
I think , by having xargs in command will add burden on tunning, since first it will add files in buffer then it will remove where as in direct command , it will keep removing once it finds the file/dir.
xargs will not burden anything. The difference between -exec and |xargs can be significant if there are a lot of arguments, with xargs being the winner. I already said this in the previous post, but let me re-iterate in more detail:
Code:
 find . -exec rm {} \;

will fork() a process for each file. If find returns a million files, you will end up with a million rm commands. This is much more expensive than doing
Code:
 find . | xargs rm

because this construct will run rm only once for many files; for how many depends on your system -- it's defined in limits.h.
Try it for yourself if you don't believe me:
Code:
$ ls | wc
  37883   37883  367719
$ time find . -maxdepth 1 -type f -exec cp {} dump \;

real    1m16.008s
user    2m0.508s
sys    0m37.818s

$ time find . -maxdepth 1 -type f | xargs cp -t dump

real    0m1.197s
user    0m0.268s
sys    0m0.712s

Newer shells also provide a '+' version of -exec, which basically does the same as xargs -- feeds the command as many arguments as it can:
Code:
 $ time find . -maxdepth 1 -type f -exec cp -t dump {} +

real    0m1.050s
user    0m0.256s
sys    0m0.660s

This User Gave Thanks to mirni For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Purging a Set of Files

Hi Frineds, I want to delete a set of files which are older than 7 days from teh current date.I am totally enw to shell scripting, can anyone help me with a sample code to list out the files which are older and then remove them from the directory. Please help THanks Viswa (5 Replies)
Discussion started by: svishh123
5 Replies

2. Shell Programming and Scripting

Error While Purging Files

find /filearchive/ -type f -mtime +7 -exec rm weblogs*.log {} \; This worked only if this comand is executed int he unix comand prompt, but when i put this in the shell script it is not recognizing the file.It says weblogs: No such file or directory Am i doing anything wrong here ? (4 Replies)
Discussion started by: svishh123
4 Replies

3. Shell Programming and Scripting

Best way to diff two huge directory trees

Hi I have a job that will be running nightly incremental backsup of a large directory tree. I did the initial backup, now I want to write a script to verify that all the files were transferred correctly. I did something like this which works in principle on small trees: diff -r -q... (6 Replies)
Discussion started by: same1290
6 Replies

4. Cybersecurity

How to fine Tune and Harden the Linux kernel

Hi, As a a security audit, how can I proceed further with Fine tuning and Hardening the linux kernel... I am not sure with the steps how to proceed further... If i do some thing wrong, then its comes with the Kernel panic error. So, I am afraid, how to do the tuning with the kernel.. (1 Reply)
Discussion started by: gsiva
1 Replies

5. Shell Programming and Scripting

Shell script for purging the 3 days old files

Hi all, I try to write shell script to the below requirement. I have Hard coded the oratab location and take the list of databases from oratab and find out archive log locations for each database, and list more than 3 days old files for each location and purge those. ... (2 Replies)
Discussion started by: mak_boop
2 Replies

6. Shell Programming and Scripting

purging of Files

Hello All, I want to delete the files based on the days. like, Files available under directory /abc want to delete if they are older than 15 days. Files available under directory /pqr want to delete if they are 7 days old and some files under directory /xyz should get deleted if they are... (5 Replies)
Discussion started by: ssachins
5 Replies

7. Programming

SQL : Fine tune Insert by query

i would like to know how can i fine tune the following query since the cost of the query is too high .. insert into temp temp_1 select a,b,c,d from xxxx .. database used is IDS.. (1 Reply)
Discussion started by: expert
1 Replies

8. Shell Programming and Scripting

Fine tune this perl script to add router

Hi, I have this routine that reads a microsoft dhcp.netsh dump. Where it finds optionvalue 3 STRING "0.0.0.0" Replace it with the router IP based on the network !/usr/bin/perl while ( <> ) { if ( /\# NET / ) { $net = $'; $net =~ s///g; } else { s/set optionvalue 3... (1 Reply)
Discussion started by: richsark
1 Replies

9. Shell Programming and Scripting

Disc space issues and purging of files

Hi All, I am looking forward to create a unix shell script to purge the files. The requirement is: 1) Do df -k and check the current space occupied for the /a1 folder. 2) If the space consumed is greater than 90 %, delete all the DEF* files from a subfolder /a1/archive. Example: df... (4 Replies)
Discussion started by: shilpa_acc
4 Replies

10. UNIX for Beginners Questions & Answers

Need help with listing file name and modified date on a huge directory

hi, We have a huge directory that ha 5.1 Million files in it. We are trying to get the file name and modified timestamp of the most recent 3 years from this huge directory for a migration project. However, the ls command (background process) to list the file names and timestamp is running for... (2 Replies)
Discussion started by: subbu
2 Replies
XARGS(1)						    BSD General Commands Manual 						  XARGS(1)

NAME
xargs -- construct argument list(s) and execute utility SYNOPSIS
xargs [-0opt] [-E eofstr] [-I replstr [-R replacements]] [-J replstr] [-L number] [-n number [-x]] [-P maxprocs] [-s size] [utility [argument ...]] DESCRIPTION
The xargs utility reads space, tab, newline and end-of-file delimited strings from the standard input and executes utility with the strings as arguments. Any arguments specified on the command line are given to utility upon each invocation, followed by some number of the arguments read from the standard input of xargs. The utility is repeatedly executed until standard input is exhausted. Spaces, tabs and newlines may be embedded in arguments using single (`` ' '') or double (``"'') quotes or backslashes (``''). Single quotes escape all non-single quote characters, excluding newlines, up to the matching single quote. Double quotes escape all non-double quote char- acters, excluding newlines, up to the matching double quote. Any single character, including newlines, may be escaped by a backslash. The options are as follows: -0 Change xargs to expect NUL (``'') characters as separators, instead of spaces and newlines. This is expected to be used in concert with the -print0 function in find(1). -E eofstr Use eofstr as a logical EOF marker. -I replstr Execute utility for each input line, replacing one or more occurrences of replstr in up to replacements (or 5 if no -R flag is speci- fied) arguments to utility with the entire line of input. The resulting arguments, after replacement is done, will not be allowed to grow beyond 255 bytes; this is implemented by concatenating as much of the argument containing replstr as possible, to the con- structed arguments to utility, up to 255 bytes. The 255 byte limit does not apply to arguments to utility which do not contain replstr, and furthermore, no replacement will be done on utility itself. Implies -x. -J replstr If this option is specified, xargs will use the data read from standard input to replace the first occurrence of replstr instead of appending that data after all other arguments. This option will not affect how many arguments will be read from input (-n), or the size of the command(s) xargs will generate (-s). The option just moves where those arguments will be placed in the command(s) that are executed. The replstr must show up as a distinct argument to xargs. It will not be recognized if, for instance, it is in the middle of a quoted string. Furthermore, only the first occurrence of the replstr will be replaced. For example, the following com- mand will copy the list of files and directories which start with an uppercase letter in the current directory to destdir: /bin/ls -1d [A-Z]* | xargs -J % cp -rp % destdir -L number Call utility for every number non-empty lines read. A line ending with a space continues to the next non-empty line. If EOF is reached and fewer lines have been read than number then utility will be called with the available lines. The -L and -n options are mutually-exclusive; the last one given will be used. -n number Set the maximum number of arguments taken from standard input for each invocation of utility. An invocation of utility will use less than number standard input arguments if the number of bytes accumulated (see the -s option) exceeds the specified size or there are fewer than number arguments remaining for the last invocation of utility. The current default value for number is 5000. -o Reopen stdin as /dev/tty in the child process before executing the command. This is useful if you want xargs to run an interactive application. -P maxprocs Parallel mode: run at most maxprocs invocations of utility at once. -p Echo each command to be executed and ask the user whether it should be executed. An affirmative response, 'y' in the POSIX locale, causes the command to be executed, any other response causes it to be skipped. No commands are executed if the process is not attached to a terminal. -R replacements Specify the maximum number of arguments that -I will do replacement in. If replacements is negative, the number of arguments in which to replace is unbounded. -s size Set the maximum number of bytes for the command line length provided to utility. The sum of the length of the utility name, the arguments passed to utility (including NULL terminators) and the current environment will be less than or equal to this number. The current default value for size is ARG_MAX - 4096. -t Echo the command to be executed to standard error immediately before it is executed. -x Force xargs to terminate immediately if a command line containing number arguments will not fit in the specified (or default) command line length. If utility is omitted, echo(1) is used. Undefined behavior may occur if utility reads from the standard input. The xargs utility exits immediately (without processing any further input) if a command line cannot be assembled, utility cannot be invoked, an invocation of utility is terminated by a signal, or an invocation of utility exits with a value of 255. LEGACY DESCRIPTION
In legacy mode, the -L option treats all newlines as end-of-line, regardless of whether the line is empty or ends with a space. In addition, the -L and -n options are not mutually-exclusive. For more information about legacy mode, see compat(5). EXIT STATUS
The xargs utility exits with a value of 0 if no error occurs. If utility cannot be found, xargs exits with a value of 127, otherwise if utility cannot be executed, xargs exits with a value of 126. If any other error occurs, xargs exits with a value of 1. SEE ALSO
echo(1), find(1), execvp(3), compat(5) STANDARDS
The xargs utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compliant. The -J, -o, -P and -R options are non-standard FreeBSD exten- sions which may not be available on other operating systems. HISTORY
The xargs command appeared in PWB UNIX. BUGS
If utility attempts to invoke another command such that the number of arguments or the size of the environment is increased, it risks execvp(3) failing with E2BIG. The xargs utility does not take multibyte characters into account when performing string comparisons for the -I and -J options, which may lead to incorrect results in some locales. BSD
August 2, 2004 BSD
All times are GMT -4. The time now is 02:53 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy