grep multiple text files in folder into 1 text file?


 
Thread Tools Search this Thread
# 1  
Question grep multiple text files in folder into 1 text file?

How do I use the grep command to take mutiple text files in a folder and make one huge text file out of them. I'm using Mac OS X and can not find a text tool that does it so I figured I'd resort to the BSD Unix CLI for a solution... there are 5,300 files that I want to write to one huge file so that I can import it into a database in one shot... any help would be geratly appreciated. Thanx.
# 2  
Why do you have to use the grep command?

cd your_folder
for i in `ls *`
do
pg $i >> all
done
# 3  
With 5,300 files, the `ls *` will probably exceed the max line length and pg is a slow way to cat a file.

Code:
#! /usr/bin/ksh
ls | while read filename ; do
      cat $filename
done > all
exit 0

will work. If the xargs program is available,

(ls * | xargs cat ) > all
will be a very fast solution.

I also wonder about the mention of grep. If you need to extract certain lines switch "cat" with "grep string" in both solutions.
# 4  
Wow - I tried
(ls * | xargs cat) > all

It works the same and it's cool that it's so small.. but I keep getting the error:

cat: input/output files 'all' identical

The large file is created like it should be, but is there something additional to get rid of / suppress this error?
# 5  
that error is just letting you know that it also tried to run on the "all" file and it failed.

nothing to worry about.
# 6  
error messages

the error message doesn't meen anything however if there are a lot of files in the folder it can be annoying to get all of those messages

you can get rid of those error messages by sending them to another file.

the command

ls * | xargs cat > all 2> /dev/null

works on my system (Sun,solaris)
it may work on yours also
# 7  
Quote:
Originally posted by oombera
Wow - I tried
(ls * | xargs cat) > all

It works the same and it's cool that it's so small.. but I keep getting the error:

cat: input/output files 'all' identical

The large file is created like it should be, but is there something additional to get rid of / suppress this error?
Ouch! First, that line has the same problem that your original solution did, we are asking the shell to replace * with a list of all files in the current directory. And that won't fly with 5,300 files. What I meant to type is:

(ls | xargs cat) > all

I also didn't think about the output file popping into existence before the ls ran. The best solution for that is something like:

(ls | xargs cat) > /some/dir/all

cat is smart enough to catch this, but grep isn't. And I'm still thinking that the op wanted to use grep to extract certain lines.
 

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #836
Difficulty: Medium
SOAP is a data serialization specification.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to add a text at the beginning of a text files in a folder?

how to add a text ( surya) at the beginning of a text files (so many) in folder text file: 111111 555555 666666 result: surya 111111 555555 666666 (3 Replies)
Discussion started by: suryanarayana
3 Replies

2. Shell Programming and Scripting

Split a text file into multiple text files?

I have a text file with entries like 1186 5556 90844 7873 7722 12 7890.6 78.52 6679 3455 9867 1127 5642 ..N so many records like this. I want to split this file into multiple files like cluster1.txt, cluster2.txt, cluster3.txt, ..... clusterN.txt. (4 Replies)
Discussion started by: sammy777
4 Replies

3. Shell Programming and Scripting

Merge the multiple text files into one file

Hi All, I am trying to merge all the text files into one file using below snippet cat /home/Temp/Test/Log/*.txt >> all.txt But it seems it is not working. I have multiple files like Output_ServerName1.txt, Output_ServreName2.txt I want to merge each file into one single file and... (6 Replies)
Discussion started by: sharsour
6 Replies

4. Shell Programming and Scripting

Extracting lines from text files in folder based on the numbers in another file

Hello, I have a file ff.txt that looks as follows *ABNA.txt 356 24 36 112 *AC24.txt 457 458 321 2 ABNA.txt and AC24.txt are the files in the folder named foo1. Based on the numbers in the ff.txt file, I want to extract the lines from the corresponding files in the foo1 folder and... (2 Replies)
Discussion started by: mohamad
2 Replies

5. UNIX for Dummies Questions & Answers

Changing text in multiple files, but with different text for each file

Hello, I have a situation where I want to change a line of text in multiple files, but the problem is that I want to change the text to something unique for each file. For example, let's say I have five files named bob.txt, joe.txt, john.txt, tom.txt, and zach.txt. Each of these files has a... (5 Replies)
Discussion started by: Scatterbrain26
5 Replies

6. UNIX for Dummies Questions & Answers

How to grep multiple lines from a text file using another text file?

I would like to use grep to select multiple lines from a text file using a single-column text file. Basically I want to only select lines from the first text file where the second column of the first text file matches the second text file. How do I go about doing that? Thanks! (5 Replies)
Discussion started by: evelibertine
5 Replies

7. Shell Programming and Scripting

Need to build Shell Script to search content of a text file into a folder consist several files

Have to read one file say sourcefile containing several words and having another folder containing several files. Now read the first word of Sourcefile & search it into the folder consisting sevral files, and create another file with result. We hhave to pick the filename of the file in which... (3 Replies)
Discussion started by: mukesh.baranwal
3 Replies

8. Shell Programming and Scripting

Loop through text file > Copy Folder > Edit XML files in bulk?

I have a text file which contains lines in this format - it contains 105 lines in total, but I'm just putting 4 here to keep it short: 58571,east_ppl_ppla_por 58788,east_pcy_hd_por 58704,east_pcy_ga_por 58697,east_pcy_pcybs_por It's called id_key.txt I have a sample folder called... (9 Replies)
Discussion started by: biscuitcreek
9 Replies

9. Shell Programming and Scripting

[bash help]Adding multiple lines of text into a specific spot into a text file

I am attempting to insert multiple lines of text into a specific place in a text file based on the lines above or below it. For example, Here is a portion of a zone file. IN NS ns1.domain.tld. IN NS ns2.domain.tld. IN ... (2 Replies)
Discussion started by: cdn_humbucker
2 Replies

10. Shell Programming and Scripting

Find all text files in folder and then copy to a new folder

Hi all, *I use Uwin and Cygwin emulator. Im trying to search for all text files in the current folder (C/Files) and its sub folders using find -depth -name "*.txt" The above command worked for me, but now I would like to copy all found text files to a new folder (C/Files/Text) with ... (4 Replies)
Discussion started by: cgkmal
4 Replies

Featured Tech Videos