Compare multiple files, and extract items that are common to ALL files only


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Compare multiple files, and extract items that are common to ALL files only
# 1  
Old 01-17-2013
Compare multiple files, and extract items that are common to ALL files only

I have this code

Code:
awk 'NR==FNR{a[$1]=$1;next} a[$1]' file1 file2

which does what I need it to do, but for only two files. I want to make it so that I can have multiple files (for example 30) and the code will return only the items that are in every single one of those files and ignore the ones that are not.

For simplicity of a big example, say I have 30 files, and all of them have the words "one, two, three, four, five" but 10 of them have additional words of "six, seven, eight, nine, ten". I want the output of my code to be

Code:
output:

one
two
three
four
five

# 2  
Old 01-17-2013
try:
Code:
awk '
!f[FILENAME] {for (i in t) if (!a[i]) delete t[i] ; delete a; f[FILENAME]=FILENAME}
NR==FNR {t[$0]=$0}
{a[$0]=$0}
END { for (i in t) if (!a[i]) delete t[i]; for (i in t) print i}
' file*

This User Gave Thanks to rdrtx1 For This Post:
# 3  
Old 01-17-2013
I tried following the code along, but I couldn't understand it or get it to work. Can you update it using these files as an example please?

Code:
file1:
one
two
three

Code:
file2:
one
two
three

Code:
file3:
one
two
three
four

Code:
file4:
one
three
five

Code:
file5:
one
three
six
eight

Code:
output:
one
three

# 4  
Old 01-17-2013
Maybe awk version does not support delete. try:
Code:
awk '
!f[FILENAME] {for (i in t) if (!a[i]) {t[i]=""} ;
   for (i in a) a[i]="";
   f[FILENAME]=FILENAME
}
NR==FNR {t[$0]=$0}
{a[$0]=$0}
END { for (i in t) if (!a[i]) t[i]=""; for (i in t) {if (t[i]) print i}}
' file*

This User Gave Thanks to rdrtx1 For This Post:
# 5  
Old 01-17-2013
not all awk's support the delete array paradigm.
the workaround is: split("",array)
# 6  
Old 01-17-2013
I'm sorry but I still cannot get it to work, would you be able to use my filenames in the example given? I am having trouble with the filename part
# 7  
Old 01-17-2013
Do you mean built-in variable FILENAME? What is the error?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Compare two files and print based on common variable value.

Hi All, i have below two files. FILE: NAME="/dev/sda" TYPE="disk" SIZE="60G" OWNER="root" GROUP="disk" MODE="brw-rw----" PKNAME="" MOUNTPOINT="" NAME="/dev/sda1" TYPE="part" SIZE="500M" OWNER="root" GROUP="disk" MODE="brw-rw----" PKNAME="/dev/sda" MOUNTPOINT="/boot" NAME="/dev/sda2"... (3 Replies)
Discussion started by: balu1234
3 Replies

2. UNIX for Beginners Questions & Answers

Issue with search and replacing multiple items in multiple files

Im having an issue when trying to replace the first column with a new set of values in multiple files. The results from the following code only replaces the files with the last set of values in val.txt. I want to replace all the files with all the values. for date in {1..31} do for val in... (1 Reply)
Discussion started by: ncwxpanther
1 Replies

3. Shell Programming and Scripting

Compare multiple files, identify common records and combine unique values into one file

Good morning all, I have a problem that is one step beyond a standard awk compare. I would like to compare three files which have several thousand records against a fourth file. All of them have a value in each row that is identical, and one value in each of those rows which may be duplicated... (1 Reply)
Discussion started by: nashton
1 Replies

4. UNIX for Dummies Questions & Answers

Extract common data out of multiple files

I am trying to extract common list of Organisms from different files For example I took 3 files and showed expected result. In real I have more than 1000 files. I am aware about the useful use of awk and grep but unaware in depth so need guidance regarding it. I want to use awk/ grep/ cut/... (7 Replies)
Discussion started by: macmath
7 Replies

5. Shell Programming and Scripting

Compare two sample files and find common

Hi I have two sample files attached here one file contain entries in one column and second file contains entries in many columns I have to match entries of first file with entries in secon d file form secon column onwards and if matches write "match" in front of it. I tried several... (11 Replies)
Discussion started by: manigrover
11 Replies

6. Shell Programming and Scripting

Compare a common field in two files and append a column from File 1 in File2

Hi Friends, I am new to Shell Scripting and need your help in the below situation. - I have two files (File 1 and File 2) and the contents of the files are mentioned below. - "Application handle" is the common field in both the files. (NOTE :- PLEASE REFER TO THE ATTACHMENT "Compare files... (2 Replies)
Discussion started by: Santoshbn
2 Replies

7. Shell Programming and Scripting

Extract common words from two/more csv files

I have two (or more, to make it generic) csv files. Each line contains words separated by comma. None of words have any space. The number of words per line is not fixed. Some may have one, and some may have 12... The number of lines per file is also not fixed. What I need is to find common words... (1 Reply)
Discussion started by: nick2011
1 Replies

8. UNIX for Dummies Questions & Answers

compare two files based on common field in unix

I have two files in UNIX. 1st file is Entity and Second File is References. 1st File has only one column named Entity ID and 2nd file has two columns Entity ID | Person ID. I want to produce a output file where entity id's are matching in both the files. Entity File 624197 624252 624264... (4 Replies)
Discussion started by: PRS
4 Replies

9. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

10. Shell Programming and Scripting

Get common lines from multiple files

FileA chr1 31237964 NP_001018494.1 PUM1 M340L chr1 31237964 NP_055491.1 PUM1 M340L chr1 33251518 NP_037543.1 AK2 H191D chr1 33251518 NP_001616.1 AK2 H191D chr1 57027345 NP_001004303.2 C1orf168 P270S FileB chr1 ... (9 Replies)
Discussion started by: genehunter
9 Replies
Login or Register to Ask a Question

Featured Tech Videos