Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Combine multiple files with common string into one new file.

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Combine multiple files with common string into one new file.
# 1  
Old 10-21-2010
Combine multiple files with common string into one new file.
I need to compile a large amount of data with a common string from individual text files throughout many directories.


An example data file is below. I want to search for the following string, "cc_sectors_1" and combine all the data from each file which contains this string, into one new output file.
Code:
cc_sectors_1_pt2.nii.gz
34 408.000000 0.582149 0.165659 0.250764 0.783992 
cc_sectors_2_pt2.nii.gz
10 120.000000 0.515655 0.140343 0.329384 0.711448 
cc_sectors_3_pt2.nii.gz
11 132.000000 0.552913 0.178691 0.266971 0.775907 
cc_sectors_4_pt2.nii.gz
15 180.000000 0.522139 0.174769 0.255384 0.728975 
cc_sectors_5_pt2.nii.gz
41 492.000000 0.511493 0.173014 0.220017 0.720323

I am assuming this will be an awk command. Any suggestions?

I appreciate any help!
Last edited by Scott; 10-25-2010 at 03:24 AM.. Reason: Please use code tags
# 2  
Old 10-22-2010
Hi,

I would use 'find' + 'grep' + 'awk'.

Something similar to this:
Code:
find . -type f -iname "*.txt" | xargs grep "cc_sectors_1" | awk -F":" '{ for (i=2;i<=NF;i++) printf "%s",$i ; printf "\n" }'

Regards,
Birei
# 3  
Old 10-25-2010
Hi birei,

Thanks for the response!

This worked to list all of the lines where the string was mentioned, but I need the entire file for each instance the string is in the doc. I also need the data ordered alphabetically by filename... not sure how to tackle this one at all.

I have also got the following script to work, but still do not know how to sort alphabetically by filename:
Code:
#!/bin/ksh

cd <to_your_directory>
file="$(pwd)/newoutput23.txt"

rm -f $file
find . -type f -print|xargs grep -l cc_sectors_1| while read myfile
do
   echo $myfile
done > $file

Thanks to anyone who takes the time to look at this!
Last edited by Scott; 10-25-2010 at 03:25 AM.. Reason: Please use code tags
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare multiple files, identify common records and combine unique values into one file

Good morning all, I have a problem that is one step beyond a standard awk compare. I would like to compare three files which have several thousand records against a fourth file. All of them have a value in each row that is identical, and one value in each of those rows which may be duplicated... (1 Reply)
Discussion started by: nashton
1 Replies

2. UNIX for Dummies Questions & Answers

Want to change common line from multiple files

Hi everyone, I've a requirement to modify an existing line which is common to multiple files. I need to replace that existing line with a new line. I've almost 900 ksh files to edit in the similar fashion in the same directory. Example: Existing Line: . $HOME/.eff.env (notice the "." at the... (3 Replies)
Discussion started by: kaleem.adil
3 Replies

3. Shell Programming and Scripting

Compare multiple files, and extract items that are common to ALL files only

I have this code awk 'NR==FNR{a=$1;next} a' file1 file2 which does what I need it to do, but for only two files. I want to make it so that I can have multiple files (for example 30) and the code will return only the items that are in every single one of those files and ignore the ones... (7 Replies)
Discussion started by: castrojc
7 Replies

4. Shell Programming and Scripting

Combine Multiple Files into Single One File One after other

I am trying to combine 4 .dat files into one single Output file Inputs are:- file123.dat, file256.dat, file378.dat & file490 Expected Output:- FileName=file1 {text from file1} EOF {blank line} FileName=file2 {text from file2} EOF {blank line} FileName=file3 {text from file3} EOF... (4 Replies)
Discussion started by: lancesunny
4 Replies

5. Emergency UNIX and Linux Support

Combine multiple Files into one big file

Hi Ppl, I have a requirement like i will be getting files of huge size daily and if the file size is so huge ,the files will be split into many parts and sent.The first file will have the header details followed by detail records and the consecutive files will have detail records and the last... (11 Replies)
Discussion started by: ganesh_248
11 Replies

6. Shell Programming and Scripting

Need help joining two files with a common string

Hi all, I have one file that is in the form: S0243K05_T7_S0243K05_|_BASS2243.C7_K05 groupVI. 88.76 S0137F20_SP6_S0137F20_|_BASS2137d.SPB2.2_C10 groupXXI 88.06 S0056F03_T7_S0056F03_|_BASS256c.C7_C02 groupXIX 85.99 S0056F03_T7_S0056F03_|_BASS256c.C7_C02 groupXIX 83.23... (3 Replies)
Discussion started by: repiv
3 Replies

7. Shell Programming and Scripting

Get common lines from multiple files

FileA chr1 31237964 NP_001018494.1 PUM1 M340L chr1 31237964 NP_055491.1 PUM1 M340L chr1 33251518 NP_037543.1 AK2 H191D chr1 33251518 NP_001616.1 AK2 H191D chr1 57027345 NP_001004303.2 C1orf168 P270S FileB chr1 ... (9 Replies)
Discussion started by: genehunter
9 Replies

8. Shell Programming and Scripting

Combine common line from 2 Huge files

Hi, I am having 2 huge files having line count more than 10million. The files look like: File 1 45905099 2059 942961505 3007 8450875165 7007 615565331 3015 9415586035 9012 9871573 5367 4415655 4011 44415539519 5361 3250659295 4001 5950718618 9367 File 2 44415539519 TQ03... (2 Replies)
Discussion started by: rochitsharma
2 Replies

9. Shell Programming and Scripting

Combine multiple string into 1 string group by certain criteria

Hi all, I am newbie in unix. Just have some doubts on how to join multiple lines into single line. I have 1 file with following contents. R96087641 HostName-kul480My This is no use any more %% E78343970 LocalPath-/app/usr/SG (Blank in this line) %% E73615740... (4 Replies)
Discussion started by: whchee
4 Replies

10. Shell Programming and Scripting

Combine multiple columns from multiple files

Hi there, I was wondering if someone can help me with this. I am trying the combine multiple columns from multiple files into one file. Example file 1: c0t0d0 c0t2d0 # hostname vgname c0t0d1 c0t2d1 # hostname vgname c0t0d2 c0t2d2 # hostname vgname c0t1d0 c0t3d0 # hostname vgname1... (5 Replies)
Discussion started by: martva
5 Replies
Login or Register to Ask a Question