Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Parsing common values across multiple files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parsing common values across multiple files
# 1  
Old 06-01-2012
Parsing common values across multiple files
Hi All,
I have multiple (5+) text files with single columns and I would like to grep the common values across all the text files and parse it to a new file. All the values are numerical. Please let me know how to do it using awk.
# 2  
Old 06-01-2012
Couple things to consider
Often easier to understand by providing data input examples and desired output.
And, include the commands you have tried so far.

Don't forget to use CodeTags around the data and commands.
# 3  
Old 06-01-2012
File1
Code:
1
2
4
5
6
7

File2
Code:
1
2
5
9
6
10

File3
Code:
1
2
5
9
7
10
11

Desired output
Code:
1
2
5

There will be more than 3 files.

---------- Post updated at 09:55 AM ---------- Previous update was at 09:46 AM ----------

what I tried
Code:
perl -ane 'print"$F[1]\n" unless $seen{$F[1]}++' file1.txt file2.txt file3.txt > output

# 4  
Old 06-01-2012
you can use
Code:
sort file1 file2 | uniq -d >> newfile

Last edited by Franklin52; 06-04-2012 at 03:33 AM.. Reason: Please use code tags
# 5  
Old 06-09-2012
Franklin,

I apologise for not following code tags. Read the rules about them and will follow them. Thanks for correcting me again.

Jay
# 6  
Old 06-09-2012
If there is no more than one occurrence of each value per file:
Code:
awk '++A[$1]==ARGC-1' file*

Last edited by Scrutinizer; 06-09-2012 at 01:03 PM..
This User Gave Thanks to Scrutinizer For This Post:
alister
# 7  
Old 06-09-2012
Quote:
Originally Posted by Scrutinizer
If there is no more than one occurrence of each value per file:
Code:
awk '++A[$1]==ARGC-1' file*

Very slick.

Regards,
Alister
Last edited by alister; 06-09-2012 at 03:14 PM.. Reason: It deserved a "very"
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Get both common and missing values from multiple files

Hi, I have 5 files with two columns. I need to merge all the 5 files based on column 1. If any of them are missing then corresponding 2nd column should be substituted by missing value. I know hoe to do this for 2 files. but how can I implement for 5 files. I tried this based on 5 files but it... (2 Replies)
Discussion started by: Diya123
2 Replies

2. Shell Programming and Scripting

Common values in 2 columns in 2 files

Hello, Suppose I have these 2 tab delimited files, where the second column in first file contains matching values from first column of the second file, I would like to get an output like this: File A 1 A 2 B 3 C File B A Apple C Cinnabon B Banana I would like... (1 Reply)
Discussion started by: Mohamed EL Hadi
1 Replies

3. Shell Programming and Scripting

Compare multiple files, identify common records and combine unique values into one file

Good morning all, I have a problem that is one step beyond a standard awk compare. I would like to compare three files which have several thousand records against a fourth file. All of them have a value in each row that is identical, and one value in each of those rows which may be duplicated... (1 Reply)
Discussion started by: nashton
1 Replies

4. UNIX for Dummies Questions & Answers

Want to change common line from multiple files

Hi everyone, I've a requirement to modify an existing line which is common to multiple files. I need to replace that existing line with a new line. I've almost 900 ksh files to edit in the similar fashion in the same directory. Example: Existing Line: . $HOME/.eff.env (notice the "." at the... (3 Replies)
Discussion started by: kaleem.adil
3 Replies

5. Shell Programming and Scripting

Find Common Values Across Two Files

Hi All, I have two files like below: File1 MYFILE_28012012_1112.txt|4 MYFILE_28012012_1113.txt|51 MYFILE_28012012_1114.txt|57 MYFILE_28012012_1115.txt|57 MYFILE_28012012_1116.txt|57 MYFILE_28012012_1117.txt|57 File2 MYFILE_28012012_1110.txt|57 MYFILE_28012012_1111.txt|57... (2 Replies)
Discussion started by: angshuman
2 Replies

6. Shell Programming and Scripting

Compare multiple files, and extract items that are common to ALL files only

I have this code awk 'NR==FNR{a=$1;next} a' file1 file2 which does what I need it to do, but for only two files. I want to make it so that I can have multiple files (for example 30) and the code will return only the items that are in every single one of those files and ignore the ones... (7 Replies)
Discussion started by: castrojc
7 Replies

7. Shell Programming and Scripting

Find common lines between multiple files

Hello everyone A few years Ago the user radoulov posted a fancy solution for a problem, which was about finding common lines (gene variation names) between multiple samples (files). The code was: awk 'END { for (R in rec) { n = split(rec, t, "/") if (n > 1) dup = dup ?... (5 Replies)
Discussion started by: bibb
5 Replies

8. UNIX for Dummies Questions & Answers

Extract common data out of multiple files

I am trying to extract common list of Organisms from different files For example I took 3 files and showed expected result. In real I have more than 1000 files. I am aware about the useful use of awk and grep but unaware in depth so need guidance regarding it. I want to use awk/ grep/ cut/... (7 Replies)
Discussion started by: macmath
7 Replies

9. Shell Programming and Scripting

Get common lines from multiple files

FileA chr1 31237964 NP_001018494.1 PUM1 M340L chr1 31237964 NP_055491.1 PUM1 M340L chr1 33251518 NP_037543.1 AK2 H191D chr1 33251518 NP_001616.1 AK2 H191D chr1 57027345 NP_001004303.2 C1orf168 P270S FileB chr1 ... (9 Replies)
Discussion started by: genehunter
9 Replies

10. UNIX for Dummies Questions & Answers

How to rename multiple files with a common suffix

Hi, There are multiple files like file1_11 file2_11 file3_11.....and so on. How to rename them such tht the suffix _11 is removed and they become file1, file2, file3. Any help is appreciated. Regards er_ashu (1 Reply)
Discussion started by: er_ashu
1 Replies
Login or Register to Ask a Question