Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Concatenate several thousand files in unix

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Concatenate several thousand files in unix
# 1  
Old 11-12-2009
Concatenate several thousand files in unix
Hello,

I would like to concatenate several thousand files (~40,000) into 1. I cannot do it with cat *.extension - it gives me an error that there are too many arguments.

Any suggestions?

Thanks,
Gussi
# 2  
Old 11-12-2009
This ?
Code:
find . -name "*.extension" -exec cat {} \; > file.new

# 3  
Old 11-12-2009
then use a normal loop
Code:
for file in *.extension
do
  cat "$file" >>new.file
done

# 4  
Old 11-13-2009
40,000 filenames probably exceeds ARG_MAX bytes on a lot of systems, so a solution with wildcards like * that globs could fail. find (as above) is not globbing.
Code:
for file in *.extension

does glob.

another way
Code:
ls | grep '\.extension' |
while read fname
do
   cat $fname
done > outputfile

Code:
getconf ARG_MAX

shows what the ARG_MAX limit is.
# 5  
Old 11-13-2009
Quote:
Originally Posted by jim mcnamara
40,000 filenames probably exceeds ARG_MAX bytes on a lot of systems, so a solution with wildcards like * that globs could fail. find (as above) is not globbing.
Code:
for file in *.extension

does glob.
not on mine. ARG_MAX=2621440. I think ls also is affected by ARG_MAX. The "usual" solution is to use xargs.
# 6  
Old 11-13-2009
ARG_MAX is in bytes and it includes any environment data passed to the exec.

Solution by ghostdog74 looks sound.

When limits are involved it always helps to state the Operating System and shell.
# 7  
Old 11-16-2009
Thank you Ghost Dog & Scrutinizer for your suggestions!

Used both solutions!

~Gussi
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Concatenate files and delete source files. Also have to add a comment.

- Concatenate files and delete source files. Also have to add a comment. - I need to concatenate 3 files which have the same characters in the beginning and have to remove those files and add a comment and the end. Example: cat REJ_FILE_ABC.txt REJ_FILE_XYZ.txt REJ_FILE_PQR.txt >... (0 Replies)
Discussion started by: eskay
0 Replies

2. UNIX for Dummies Questions & Answers

Concatenate files

Hi I am trying to learn linux step by step an i am wondering can i use cat command for concatenate files but i want to place context of file1 to a specific position in file2 place of file 2 and not at the end as it dose on default? Thank you. (3 Replies)
Discussion started by: iliya24
3 Replies

3. UNIX for Dummies Questions & Answers

Concatenate Several Files to One

Hi All, Need your help. I will need to concatenate around 100 files but each end of the file I will need to insert my name DIRT1228 on each of the file and before the next file is added and arrived with just one file for all the 100files. Appreciate your time. Dirt (6 Replies)
Discussion started by: dirt1228
6 Replies

4. Shell Programming and Scripting

Concatenate files

I have a file named "file1" which has the following data 10000 20000 30000 And I have a file named "file2" which has the following data ABC DEF XYZ My output should be 10000ABC 20000DEF (3 Replies)
Discussion started by: bobby1015
3 Replies

5. UNIX for Dummies Questions & Answers

Concatenate more than 200 files

Hi, I have a bunch of files (around 500) with the .log extension and each having a different name. Is there an easy way to concatenate them all into a single file? I've tried: cat *.log > complete.log and it seems that files are overlapping each other...thus the new file is not complete.... (2 Replies)
Discussion started by: amarn
2 Replies

6. Shell Programming and Scripting

Concatenate files

Hi, I want to create a batch(bash) file to combine 23 files together. These files have the same extension. I want the final file is save to a given folder. Once it is done it will delete the 23 files. Thanks for help. Need script. (6 Replies)
Discussion started by: zhshqzyc
6 Replies

7. Shell Programming and Scripting

Concatenate files

I have directory structure sales_only under which i have multiple directories for each dealer example: ../../../Sales_Only/xxx_Dealer ../../../Sales_Only/yyy_Dealer ../../../Sales_Only/zzz_Dealer Every day i have one file produce under each directory when the process runs. The requirement... (3 Replies)
Discussion started by: mohanmuthu
3 Replies

8. Infrastructure Monitoring

How to concatenate this simple line in Unix?

Hi: I use the snmpget command everyday. And Im getting tire of writing the same line evertime I have to verify something. Example of the line: snmpget -c DreamTeam dal2-hr2 ifAlias.227 The .227 its the circuit interface and also its variable; could be any other number depending on the... (13 Replies)
Discussion started by: javygonx
13 Replies

9. UNIX for Advanced & Expert Users

Concatenate text of two files in UNIX.

I have a file MyTest.csv saved in Unicode format in Incoming directory, and I have another file called MyTest.csv saved in ANSII format in InProcess Directory. I need to concatenate the text of both these files and put in another file MyTest.csv which is placed in the root directory. How do I... (1 Reply)
Discussion started by: Uniq
1 Replies

10. UNIX for Dummies Questions & Answers

How to concatenate all files.

Hi, I'm totally new to Unix. I'm an MVS mainframer but ran into a situation where a Unix server I have available will help me. I want to be able to remotely connect to another server using FTP, login and MGET all files from it's root or home directory, logout, then login as a different user and do... (1 Reply)
Discussion started by: s80bob
1 Replies
Login or Register to Ask a Question