Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Find common lines between all of the files in one folder

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Find common lines between all of the files in one folder
# 15  
Old 03-09-2018
I don't quite follow your sample files and the "desired" output.
Say take file1 - the desired result is:
Code:
54   78   79   90  344  3399
223  299  388  455  477   566

but 54 78 79 90 344 3399 is common ONLY in 2 files out of sampled 3...
Please clarify or provide a better matching outcome.
# 16  
Old 03-10-2018
a question to RudiC
Hi! Thank you for your help!
This code

Code:
awk '{CNT[$0]++; FN[$0] = FN[$0] FILENAME "-"} END {for (c in CNT) if (CNT[c]>1) {print c >> FN[c]; close (FN[c])}} ' file[123]

works really well if the filenames are file1 file2 file3 etc. But how should I use it if the filenames look like that:

Code:
AC-FOUR-136-ZEL2-ZECO-111
AC-SEVEN-56-ZEL4-ZECO-68
AC-NINE-994-ZEL3-ZECO-811
AC-ONE-4-ZEL1-ZECO-544
AC-NINE-4-53-ZEL3-ZECO-811
AC-ELEVEN-66-788-ZEL4-ZECO-87
AC-TWO-32-7788-ZEL4-ZECO-95
AC-SIX-56-111-ZEL4-ZECO-87
AC-FOURTEEN-59-1561-ZEL2-ZECO-5

Is this line a part of the code?
Code:
cf file?-*

---------- Post updated at 11:28 AM ---------- Previous update was at 11:18 AM ----------

Text lines in my files don't usually duplicates in more than two of the files, only sometimes a text line is present in three or four files.


Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!
Last edited by RudiC; 03-10-2018 at 01:24 PM.. Reason: Added CODE tags.
# 17  
Old 03-10-2018
Quote:
Originally Posted by Eve
. . .
But how should I use it if the filenames look like that:
Code:
AC-FOUR-136-ZEL2-ZECO-111
AC-SEVEN-56-ZEL4-ZECO-68
.
.
.
AC-FOURTEEN-59-1561-ZEL2-ZECO-5

How about awk ' ... ' AC-*?

Quote:
Is this line a part of the code?
Code:
cf file?-*

It is my own shell function for cat files (with wildcards)

Quote:
Text lines in my files don't usually duplicates in more than two of the files, only sometimes a text line is present in three or four files.
And what be the result of those?
This User Gave Thanks to RudiC For This Post:
Eve
# 18  
Old 03-10-2018
Thank you RudiC!
Thank you RudiC! Everything works great! Now everything is solved for me!
# 19  
Old 03-13-2018
The following variant uses less memory,
and it more efficiently writes in the END section (but uses more file handles).
One can adapt the separator character in the BEGIN section
Code:
awk 'BEGIN {sep=","} {FN[$0]=(FN[$0]=="" ? FILENAME : (FN[$0] sep FILENAME))} END {for (c in FN) if (index(FN[c],sep)) {print c > FN[c]}} ' AC-*

This User Gave Thanks to MadeInGermany For This Post:
Eve
# 20  
Old 03-13-2018
Thank you MadeInGermany!
Thank you MadeInGermany! Your code works really well too!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash to trim folder and files within a path that share a common file extension

The bash will trim the folder to trim folder. Within each of the folders (there may be more than 1) and the format is always the same, are several .bam and matching .bam.bai files (file structure) and the bashunder that executes and trims the .bam as expected but repeats the.bam.bai extentions... (9 Replies)
Discussion started by: cmccabe
9 Replies

2. Shell Programming and Scripting

Find common lines with one file and with all of the files in another folder

Hi! I would like to comm -12 with one file and with all of the files in another folder that has a 100 files or more (that file is not in that folder) to find common text lines. I would like to have each case that they have common lines to be written to a different output file and the names of the... (6 Replies)
Discussion started by: Eve
6 Replies

3. Shell Programming and Scripting

Shell Script to find common lines and replace next line

I want to find common line in two files and replace the next line of first file with the next line of second file. (sed,awk,perl,bash any solution is welcomed ) Case Ignored. Multiple Occurrence of same line. File 1: hgacdavd sndm,ACNMSDC msgid "Rome" msgstr "" kgcksdcgfkdsb... (4 Replies)
Discussion started by: madira
4 Replies

4. UNIX for Dummies Questions & Answers

Filter lines common in two files

Thanks everyone. I got that problem solved. I require one more help here. (Yes, UNIX definitely seems to be fun and useful, and I WILL eventually learn it for myself. But I am now on a different project and don't really have time to go through all the basics. So, I will really appreciate some... (6 Replies)
Discussion started by: latsyrc
6 Replies

5. Shell Programming and Scripting

Find common lines between multiple files

Hello everyone A few years Ago the user radoulov posted a fancy solution for a problem, which was about finding common lines (gene variation names) between multiple samples (files). The code was: awk 'END { for (R in rec) { n = split(rec, t, "/") if (n > 1) dup = dup ?... (5 Replies)
Discussion started by: bibb
5 Replies

6. Shell Programming and Scripting

Common lines from files

Hello guys, I need a script to get the common lines from two files with a criteria that if the first two columns match then I keep the maximum value of the 5th column.(tab separated columns) . 3rd and 4th columns corresponds to the row which has highest value for the 5th column. Sample... (2 Replies)
Discussion started by: jaysean
2 Replies

7. Shell Programming and Scripting

Common lines from files

Hello guys, I need a script to get the common lines from two files with a criteria that if the first two columns match then I keep the maximum value of the 3rd column.(tab separated columns) Sample input: file1: 111 222 0.1 333 444 0.5 555 666 0.4 file 2: 111 222 0.7 555 666... (5 Replies)
Discussion started by: jaysean
5 Replies

8. Shell Programming and Scripting

Find all text files in folder and then copy to a new folder

Hi all, *I use Uwin and Cygwin emulator. I´m trying to search for all text files in the current folder (C/Files) and its sub folders using find -depth -name "*.txt" The above command worked for me, but now I would like to copy all found text files to a new folder (C/Files/Text) with ... (4 Replies)
Discussion started by: cgkmal
4 Replies

9. UNIX for Dummies Questions & Answers

find common lines using just one column to compare and result with all columns

Hi. If we have this file A B C 7 8 9 1 2 10 and this other file A C D F 7 9 2 3 9 2 3 4 The result i´m looking for is intersection with A B C D F so the answer here will be (10 Replies)
Discussion started by: alcalina
10 Replies

10. Shell Programming and Scripting

To find all common lines from 'n' no. of files

Hi, I have one situation. I have some 6-7 no. of files in one directory & I have to extract all the lines which exist in all these files. means I need to extract all common lines from all these files & put them in a separate file. Please help. I know it could be done with the help of... (11 Replies)
Discussion started by: The Observer
11 Replies
Login or Register to Ask a Question