To find all common lines from 'n' no. of files


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting To find all common lines from 'n' no. of files
# 1  
To find all common lines from 'n' no. of files

Hi,

I have one situation. I have some 6-7 no. of files in one directory & I have to extract all the lines which exist in all these files. means I need to extract all common lines from all these files & put them in a separate file.

Please help. I know it could be done with the help of cut,sort & uniq commands. But it will take more time whenever the script is executed. I want some quick & shortcut method.

I am using ksh shell.
# 2  
Code:
cat * | sort > /tmp/alllines
uniq -d /tmp/alllines > /tmp/repeatedlines

-mschwage
# 3  
Sorry Sir,
My requirement is I want the lines which are present in all these 'n' files. Means every line which will come in output must be present in each of these 'n' files.

Please guide.

Last edited by The Observer; 06-13-2008 at 08:07 AM..
# 4  
Try this one

>comm_data.txt
for fname in 1.txt 2.txt 3.txt 4.txt
do
if [ ! -s comm_data.txt ]
then
sort $fname > comm_data.txt
else
sort $fname > tmp2
comm -12 comm_data.txt tmp2 > tmp3
mv tmp3 comm_data.txt
fi
done
rm -f tmp2

If the contents of ur files are in order you can use following

>comm_data.txt
for fname in 1.txt 2.txt 3.txt 4.txt
do
if [ ! -s comm_data.txt ]
then
cp $fname comm_data.txt
else
comm -12 comm_data.txt $fname > tmp3
mv tmp3 comm_data.txt
fi
done
cat comm_data.txt
# 5  
If you have three files f1 f2 f3:

Code:
cat f1 f2 f3 | awk '{a[$0]++} END{for (i in a) if (a[i]==3) print i}'

# 6  
But ....

Thanks ripat, it's perfectly working fine. But ,I have a little concern.
Actually, I won't be sure how many no. of files will get generated everytime the script runs. So, I will be storing the no. of files in a variable & in the code -

cat f1 f2 f3 | awk '{a[$0]++} END{for (i in a) if (a[i]==3) print i}'

I think I have to use the value in that variable in place of 3 here. I tried to replace 3 with variable here. But seems to be not working. I even attempted 'awk -v' option. But, in vain.

Can u please quickly help me out in this. The requirement is quite urgent.

Thanks in advance for ur help.
# 7  
Code:
ls -1 | wc -l | read filecnt
awk -v filecnt=$filecnt '{a[$0]++} END{for (i in a) if (a[i]==filecnt) print i}'

Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #64
Difficulty: Easy
HTTP uses UDP as the core transport protocol on the Internet.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Find common lines between all of the files in one folder

Could it be possible to find common lines between all of the files in one folder? Just like comm -12 . So all of the files two at a time. I would like all of the outcomes to be written to a different files, and the file names could be simply numbers - 1 , 2 , 3 etc. All of the file names contain... (19 Replies)
Discussion started by: Eve
19 Replies

2. Shell Programming and Scripting

Find common lines with one file and with all of the files in another folder

Hi! I would like to comm -12 with one file and with all of the files in another folder that has a 100 files or more (that file is not in that folder) to find common text lines. I would like to have each case that they have common lines to be written to a different output file and the names of the... (6 Replies)
Discussion started by: Eve
6 Replies

3. Shell Programming and Scripting

Find common files between two directories

I have two directories Dir 1 /home/sid/release1 Dir 2 /home/sid/release2 I want to find the common files between the two directories Dir 1 files /home/sid/release1>ls -lrt total 16 -rw-r--r-- 1 sid cool 0 Jun 19 12:53 File123 -rw-r--r-- 1 sid cool 0 Jun 19 12:53... (5 Replies)
Discussion started by: sidnow
5 Replies

4. Shell Programming and Scripting

Shell Script to find common lines and replace next line

I want to find common line in two files and replace the next line of first file with the next line of second file. (sed,awk,perl,bash any solution is welcomed ) Case Ignored. Multiple Occurrence of same line. File 1: hgacdavd sndm,ACNMSDC msgid "Rome" msgstr "" kgcksdcgfkdsb... (4 Replies)
Discussion started by: madira
4 Replies

5. UNIX for Dummies Questions & Answers

Filter lines common in two files

Thanks everyone. I got that problem solved. I require one more help here. (Yes, UNIX definitely seems to be fun and useful, and I WILL eventually learn it for myself. But I am now on a different project and don't really have time to go through all the basics. So, I will really appreciate some... (6 Replies)
Discussion started by: latsyrc
6 Replies

6. Shell Programming and Scripting

Find common lines between multiple files

Hello everyone A few years Ago the user radoulov posted a fancy solution for a problem, which was about finding common lines (gene variation names) between multiple samples (files). The code was: awk 'END { for (R in rec) { n = split(rec, t, "/") if (n > 1) dup = dup ?... (5 Replies)
Discussion started by: bibb
5 Replies

7. Shell Programming and Scripting

Get common lines from multiple files

FileA chr1 31237964 NP_001018494.1 PUM1 M340L chr1 31237964 NP_055491.1 PUM1 M340L chr1 33251518 NP_037543.1 AK2 H191D chr1 33251518 NP_001616.1 AK2 H191D chr1 57027345 NP_001004303.2 C1orf168 P270S FileB chr1 ... (9 Replies)
Discussion started by: genehunter
9 Replies

8. Shell Programming and Scripting

Common lines from files

Hello guys, I need a script to get the common lines from two files with a criteria that if the first two columns match then I keep the maximum value of the 5th column.(tab separated columns) . 3rd and 4th columns corresponds to the row which has highest value for the 5th column. Sample... (2 Replies)
Discussion started by: jaysean
2 Replies

9. Shell Programming and Scripting

Common lines from files

Hello guys, I need a script to get the common lines from two files with a criteria that if the first two columns match then I keep the maximum value of the 3rd column.(tab separated columns) Sample input: file1: 111 222 0.1 333 444 0.5 555 666 0.4 file 2: 111 222 0.7 555 666... (5 Replies)
Discussion started by: jaysean
5 Replies

10. UNIX for Dummies Questions & Answers

find common lines using just one column to compare and result with all columns

Hi. If we have this file A B C 7 8 9 1 2 10 and this other file A C D F 7 9 2 3 9 2 3 4 The result im looking for is intersection with A B C D F so the answer here will be (10 Replies)
Discussion started by: alcalina
10 Replies

Featured Tech Videos