Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Concatenate several thousand files in unix Post 302371039 by ghostdog74 on Friday 13th of November 2009 07:41:57 AM
Old 11-13-2009
Quote:
Originally Posted by jim mcnamara
40,000 filenames probably exceeds ARG_MAX bytes on a lot of systems, so a solution with wildcards like * that globs could fail. find (as above) is not globbing.
Code:
for file in *.extension

does glob.
not on mine. ARG_MAX=2621440. I think ls also is affected by ARG_MAX. The "usual" solution is to use xargs.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to concatenate all files.

Hi, I'm totally new to Unix. I'm an MVS mainframer but ran into a situation where a Unix server I have available will help me. I want to be able to remotely connect to another server using FTP, login and MGET all files from it's root or home directory, logout, then login as a different user and do... (1 Reply)
Discussion started by: s80bob
1 Replies

2. UNIX for Advanced & Expert Users

Concatenate text of two files in UNIX.

I have a file MyTest.csv saved in Unicode format in Incoming directory, and I have another file called MyTest.csv saved in ANSII format in InProcess Directory. I need to concatenate the text of both these files and put in another file MyTest.csv which is placed in the root directory. How do I... (1 Reply)
Discussion started by: Uniq
1 Replies

3. Infrastructure Monitoring

How to concatenate this simple line in Unix?

Hi: I use the snmpget command everyday. And Im getting tire of writing the same line evertime I have to verify something. Example of the line: snmpget -c DreamTeam dal2-hr2 ifAlias.227 The .227 its the circuit interface and also its variable; could be any other number depending on the... (13 Replies)
Discussion started by: javygonx
13 Replies

4. Shell Programming and Scripting

Concatenate files

I have directory structure sales_only under which i have multiple directories for each dealer example: ../../../Sales_Only/xxx_Dealer ../../../Sales_Only/yyy_Dealer ../../../Sales_Only/zzz_Dealer Every day i have one file produce under each directory when the process runs. The requirement... (3 Replies)
Discussion started by: mohanmuthu
3 Replies

5. Shell Programming and Scripting

Concatenate files

Hi, I want to create a batch(bash) file to combine 23 files together. These files have the same extension. I want the final file is save to a given folder. Once it is done it will delete the 23 files. Thanks for help. Need script. (6 Replies)
Discussion started by: zhshqzyc
6 Replies

6. UNIX for Dummies Questions & Answers

Concatenate more than 200 files

Hi, I have a bunch of files (around 500) with the .log extension and each having a different name. Is there an easy way to concatenate them all into a single file? I've tried: cat *.log > complete.log and it seems that files are overlapping each other...thus the new file is not complete.... (2 Replies)
Discussion started by: amarn
2 Replies

7. Shell Programming and Scripting

Concatenate files

I have a file named "file1" which has the following data 10000 20000 30000 And I have a file named "file2" which has the following data ABC DEF XYZ My output should be 10000ABC 20000DEF (3 Replies)
Discussion started by: bobby1015
3 Replies

8. UNIX for Dummies Questions & Answers

Concatenate Several Files to One

Hi All, Need your help. I will need to concatenate around 100 files but each end of the file I will need to insert my name DIRT1228 on each of the file and before the next file is added and arrived with just one file for all the 100files. Appreciate your time. Dirt (6 Replies)
Discussion started by: dirt1228
6 Replies

9. UNIX for Dummies Questions & Answers

Concatenate files

Hi I am trying to learn linux step by step an i am wondering can i use cat command for concatenate files but i want to place context of file1 to a specific position in file2 place of file 2 and not at the end as it dose on default? Thank you. (3 Replies)
Discussion started by: iliya24
3 Replies

10. UNIX for Dummies Questions & Answers

Concatenate files and delete source files. Also have to add a comment.

- Concatenate files and delete source files. Also have to add a comment. - I need to concatenate 3 files which have the same characters in the beginning and have to remove those files and add a comment and the end. Example: cat REJ_FILE_ABC.txt REJ_FILE_XYZ.txt REJ_FILE_PQR.txt >... (0 Replies)
Discussion started by: eskay
0 Replies
XARGS(1)						    BSD General Commands Manual 						  XARGS(1)

NAME
xargs -- construct argument list(s) and execute utility SYNOPSIS
xargs [-0opt] [-E eofstr] [-I replstr [-R replacements]] [-J replstr] [-L number] [-n number [-x]] [-P maxprocs] [-s size] [utility [argument ...]] DESCRIPTION
The xargs utility reads space, tab, newline and end-of-file delimited strings from the standard input and executes utility with the strings as arguments. Any arguments specified on the command line are given to utility upon each invocation, followed by some number of the arguments read from the standard input of xargs. The utility is repeatedly executed until standard input is exhausted. Spaces, tabs and newlines may be embedded in arguments using single (`` ' '') or double (``"'') quotes or backslashes (``''). Single quotes escape all non-single quote characters, excluding newlines, up to the matching single quote. Double quotes escape all non-double quote char- acters, excluding newlines, up to the matching double quote. Any single character, including newlines, may be escaped by a backslash. The options are as follows: -0 Change xargs to expect NUL (``'') characters as separators, instead of spaces and newlines. This is expected to be used in concert with the -print0 function in find(1). -E eofstr Use eofstr as a logical EOF marker. -I replstr Execute utility for each input line, replacing one or more occurrences of replstr in up to replacements (or 5 if no -R flag is speci- fied) arguments to utility with the entire line of input. The resulting arguments, after replacement is done, will not be allowed to grow beyond 255 bytes; this is implemented by concatenating as much of the argument containing replstr as possible, to the con- structed arguments to utility, up to 255 bytes. The 255 byte limit does not apply to arguments to utility which do not contain replstr, and furthermore, no replacement will be done on utility itself. Implies -x. -J replstr If this option is specified, xargs will use the data read from standard input to replace the first occurrence of replstr instead of appending that data after all other arguments. This option will not affect how many arguments will be read from input (-n), or the size of the command(s) xargs will generate (-s). The option just moves where those arguments will be placed in the command(s) that are executed. The replstr must show up as a distinct argument to xargs. It will not be recognized if, for instance, it is in the middle of a quoted string. Furthermore, only the first occurrence of the replstr will be replaced. For example, the following com- mand will copy the list of files and directories which start with an uppercase letter in the current directory to destdir: /bin/ls -1d [A-Z]* | xargs -J % cp -rp % destdir -L number Call utility for every number non-empty lines read. A line ending with a space continues to the next non-empty line. If EOF is reached and fewer lines have been read than number then utility will be called with the available lines. The -L and -n options are mutually-exclusive; the last one given will be used. -n number Set the maximum number of arguments taken from standard input for each invocation of utility. An invocation of utility will use less than number standard input arguments if the number of bytes accumulated (see the -s option) exceeds the specified size or there are fewer than number arguments remaining for the last invocation of utility. The current default value for number is 5000. -o Reopen stdin as /dev/tty in the child process before executing the command. This is useful if you want xargs to run an interactive application. -P maxprocs Parallel mode: run at most maxprocs invocations of utility at once. -p Echo each command to be executed and ask the user whether it should be executed. An affirmative response, 'y' in the POSIX locale, causes the command to be executed, any other response causes it to be skipped. No commands are executed if the process is not attached to a terminal. -R replacements Specify the maximum number of arguments that -I will do replacement in. If replacements is negative, the number of arguments in which to replace is unbounded. -s size Set the maximum number of bytes for the command line length provided to utility. The sum of the length of the utility name, the arguments passed to utility (including NULL terminators) and the current environment will be less than or equal to this number. The current default value for size is ARG_MAX - 4096. -t Echo the command to be executed to standard error immediately before it is executed. -x Force xargs to terminate immediately if a command line containing number arguments will not fit in the specified (or default) command line length. If utility is omitted, echo(1) is used. Undefined behavior may occur if utility reads from the standard input. The xargs utility exits immediately (without processing any further input) if a command line cannot be assembled, utility cannot be invoked, an invocation of utility is terminated by a signal, or an invocation of utility exits with a value of 255. LEGACY DESCRIPTION
In legacy mode, the -L option treats all newlines as end-of-line, regardless of whether the line is empty or ends with a space. In addition, the -L and -n options are not mutually-exclusive. For more information about legacy mode, see compat(5). EXIT STATUS
The xargs utility exits with a value of 0 if no error occurs. If utility cannot be found, xargs exits with a value of 127, otherwise if utility cannot be executed, xargs exits with a value of 126. If any other error occurs, xargs exits with a value of 1. SEE ALSO
echo(1), find(1), execvp(3), compat(5) STANDARDS
The xargs utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compliant. The -J, -o, -P and -R options are non-standard FreeBSD exten- sions which may not be available on other operating systems. HISTORY
The xargs command appeared in PWB UNIX. BUGS
If utility attempts to invoke another command such that the number of arguments or the size of the environment is increased, it risks execvp(3) failing with E2BIG. The xargs utility does not take multibyte characters into account when performing string comparisons for the -I and -J options, which may lead to incorrect results in some locales. BSD
August 2, 2004 BSD
All times are GMT -4. The time now is 08:34 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy