Find common lines between all of the files in one folder Post: 303014324

Sponsored Content

Top Forums Shell Programming and Scripting Find common lines between all of the files in one folder Post 303014324 by Eve on Friday 9th of March 2018 09:25:07 AM

03-09-2018

Registered User

Hi!

Here are three examples of the contents of the files:

file1

Code:

2   78   99  129  665   765
   3   88   99  543  876   988
   7   45   54   99  120   987
  13   23  167  334 2378  8765
  15   17   18 1125 2356  6765
  54   78   79   90  344  3399
 111  233  788  999 3421  7654
 223  299  388  455  477   566

file2

Code:

3   22   78   87  773   876
   4    9   77  890  977  7655
   7    8   23  854 1276  3343
  33  122  665  888  997   999
  54   78   79   90  344  3399
 223  299  388  455  477   566
 228  332  339  453  988  1299

file3

Code:

1  112  134  235  734  1123
   5   35   84   98 1889  2300
   7    8   23  854 1276  3343
  15   17   18 1125 2356  6765
  45  443  556  887  889   987
 111  233  788  999 3421  7654

And the desired outcome would be three files with one file containg these two lines

Code:

54   78   79   90  344  3399
 223  299  388  455  477   566

and the second file containing these two lines

Code:

15   17   18 1125 2356  6765
 111  233  788  999 3421  7654

and the third file containing this lines

Code:

7    8   23  854 1276  3343

The file names look like this

Code:

AC-FOUR-136-ZEL2-ZECO-111
AC-SEVEN-56-ZEL4-ZECO-68
AC-NINE-994-ZEL3-ZECO-811
AC-ONE-4-ZEL1-ZECO-544
AC-NINE-4-53-ZEL3-ZECO-811
AC-ELEVEN-66-788-ZEL4-ZECO-87
AC-TWO-32-7788-ZEL4-ZECO-95
AC-SIX-56-111-ZEL4-ZECO-87
AC-FOURTEEN-59-1561-ZEL2-ZECO-5

I have to work with 100-200 new files every week to find common lines two at a time. The files contain somewhere between 1000-100000 lines. The examples above are of course a lot shorter.

This code unfortunately didn't help

Code:

awk '{ sub(/[[:space:]]+$/, "") } ++cnt[$0]==2' * > /tmp/outfile

I hope this post is helpful!

Last edited by vgersh99; 03-09-2018 at 10:43 AM.. Reason: code tags, please!

Eve

View Public Profile for Eve

Find all posts by Eve

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

To find all common lines from 'n' no. of files

Hi, I have one situation. I have some 6-7 no. of files in one directory & I have to extract all the lines which exist in all these files. means I need to extract all common lines from all these files & put them in a separate file. Please help. I know it could be done with the help of...

2. UNIX for Dummies Questions & Answers

find common lines using just one column to compare and result with all columns

Hi. If we have this file A B C 7 8 9 1 2 10 and this other file A C D F 7 9 2 3 9 2 3 4 The result i�m looking for is intersection with A B C D F so the answer here will be

3. Shell Programming and Scripting

Find all text files in folder and then copy to a new folder

Hi all, *I use Uwin and Cygwin emulator. I�m trying to search for all text files in the current folder (C/Files) and its sub folders using find -depth -name "*.txt" The above command worked for me, but now I would like to copy all found text files to a new folder (C/Files/Text) with ...

4. Shell Programming and Scripting

Common lines from files

Hello guys, I need a script to get the common lines from two files with a criteria that if the first two columns match then I keep the maximum value of the 3rd column.(tab separated columns) Sample input: file1: 111 222 0.1 333 444 0.5 555 666 0.4 file 2: 111 222 0.7 555 666...

5. Shell Programming and Scripting

Common lines from files

Hello guys, I need a script to get the common lines from two files with a criteria that if the first two columns match then I keep the maximum value of the 5th column.(tab separated columns) . 3rd and 4th columns corresponds to the row which has highest value for the 5th column. Sample...

6. Shell Programming and Scripting

Find common lines between multiple files

Hello everyone A few years Ago the user radoulov posted a fancy solution for a problem, which was about finding common lines (gene variation names) between multiple samples (files). The code was: awk 'END { for (R in rec) { n = split(rec, t, "/") if (n > 1) dup = dup ?...

7. UNIX for Dummies Questions & Answers

Filter lines common in two files

Thanks everyone. I got that problem solved. I require one more help here. (Yes, UNIX definitely seems to be fun and useful, and I WILL eventually learn it for myself. But I am now on a different project and don't really have time to go through all the basics. So, I will really appreciate some...

8. Shell Programming and Scripting

Shell Script to find common lines and replace next line

I want to find common line in two files and replace the next line of first file with the next line of second file. (sed,awk,perl,bash any solution is welcomed ) Case Ignored. Multiple Occurrence of same line. File 1: hgacdavd sndm,ACNMSDC msgid "Rome" msgstr "" kgcksdcgfkdsb...

9. Shell Programming and Scripting

Find common lines with one file and with all of the files in another folder

Hi! I would like to comm -12 with one file and with all of the files in another folder that has a 100 files or more (that file is not in that folder) to find common text lines. I would like to have each case that they have common lines to be written to a different output file and the names of the...

10. Shell Programming and Scripting

Bash to trim folder and files within a path that share a common file extension

The bash will trim the folder to trim folder. Within each of the folders (there may be more than 1) and the format is always the same, are several .bam and matching .bam.bai files (file structure) and the bashunder that executes and trims the .bam as expected but repeats the.bam.bai extentions...

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

To find all common lines from 'n' no. of files

Discussion started by: The Observer

2. UNIX for Dummies Questions & Answers

find common lines using just one column to compare and result with all columns

Discussion started by: alcalina

3. Shell Programming and Scripting

Find all text files in folder and then copy to a new folder

Discussion started by: cgkmal

4. Shell Programming and Scripting

Common lines from files

Discussion started by: jaysean

5. Shell Programming and Scripting

Common lines from files

Discussion started by: jaysean

6. Shell Programming and Scripting

Find common lines between multiple files

Discussion started by: bibb

7. UNIX for Dummies Questions & Answers

Filter lines common in two files

Discussion started by: latsyrc

8. Shell Programming and Scripting

Shell Script to find common lines and replace next line

Discussion started by: madira

9. Shell Programming and Scripting

Find common lines with one file and with all of the files in another folder

Discussion started by: Eve

10. Shell Programming and Scripting

Bash to trim folder and files within a path that share a common file extension

Discussion started by: cmccabe