Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: To find all common lines from 'n' no. of files
Top Forums Shell Programming and Scripting To find all common lines from 'n' no. of files Post 302205871 by drl on Monday 16th of June 2008 01:03:44 PM
Old 06-16-2008
Hi.

I ran both awk solutions and they seemed to work. There is one aspect that may be troubling. If the files contain no duplicates, then all is well. However, here is an example where the trouble might occur. I am using radoulov's code since it is a bit shorter:
Code:
#!/usr/bin/env sh

# @(#) user2    Demonstrate finding lines in common.

#  ____
# /
# |   Infrastructure BEGIN

set -o nounset
echo

## The shebang using "env" line is designed for portability. For
#  higher security, use:
#
#  #!/bin/sh -

## Use local command version for the commands in this demonstration.

set +o nounset
echo "(Versions displayed with local utility \"version\")"
version >/dev/null 2>&1 && version =o $(_eat $0 $1) awk
set -o nounset

for file in f*
do
  echo
  echo " -- $file --"
  cat -n $file
done

# Use nawk or /usr/xpg4/bin/awk on Solaris.

# |   Infrastructure END
# \
#  ---

echo
echo " Results from awk:"
filecnt=$( ls -1 f* | wc -l )

awk '
END     { for (r in _)
                if (_[r] == ARGC - 1)
                        print r
        }
        { _[$0]++ }
' f*

exit 0

Producing:
Code:
% ./user2

(Versions displayed with local utility "version")
Linux 2.6.11-x1
GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu)
GNU Awk 3.1.4

 -- f1 --
     1  a
     2  b
     3  x
     4  x

 -- f2 --
     1  a
     2  d
     3  y
     4  y

 Results from awk:
x
y
a

Note that "x" and "y" are not common to the files, only "a". In cases like this, more work would be necessary to ensure that a line was common to all files, and not simply replicated the appropriate number of times in total among some of the files ... cheers, drl
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

how to find common words and take them out from two files

Hi, everyone, Let's say, we have xxx.txt A 1 2 3 4 5 C 1 2 3 4 5 E 1 2 3 4 5 yyy.txt A 1 2 3 4 5 B 1 2 3 4 5 C 1 2 3 4 5 D 1 2 3 4 5 E 1 2 3 4 5 First I match the first column I find intersection (A,C, E), then I want to take those lines with ACE out from yyy.txt, like A 1... (11 Replies)
Discussion started by: kaixinsjtu
11 Replies

2. UNIX for Dummies Questions & Answers

find common lines using just one column to compare and result with all columns

Hi. If we have this file A B C 7 8 9 1 2 10 and this other file A C D F 7 9 2 3 9 2 3 4 The result i´m looking for is intersection with A B C D F so the answer here will be (10 Replies)
Discussion started by: alcalina
10 Replies

3. Shell Programming and Scripting

Common lines from files

Hello guys, I need a script to get the common lines from two files with a criteria that if the first two columns match then I keep the maximum value of the 3rd column.(tab separated columns) Sample input: file1: 111 222 0.1 333 444 0.5 555 666 0.4 file 2: 111 222 0.7 555 666... (5 Replies)
Discussion started by: jaysean
5 Replies

4. Shell Programming and Scripting

Common lines from files

Hello guys, I need a script to get the common lines from two files with a criteria that if the first two columns match then I keep the maximum value of the 5th column.(tab separated columns) . 3rd and 4th columns corresponds to the row which has highest value for the 5th column. Sample... (2 Replies)
Discussion started by: jaysean
2 Replies

5. Shell Programming and Scripting

Get common lines from multiple files

FileA chr1 31237964 NP_001018494.1 PUM1 M340L chr1 31237964 NP_055491.1 PUM1 M340L chr1 33251518 NP_037543.1 AK2 H191D chr1 33251518 NP_001616.1 AK2 H191D chr1 57027345 NP_001004303.2 C1orf168 P270S FileB chr1 ... (9 Replies)
Discussion started by: genehunter
9 Replies

6. Shell Programming and Scripting

Find common lines between multiple files

Hello everyone A few years Ago the user radoulov posted a fancy solution for a problem, which was about finding common lines (gene variation names) between multiple samples (files). The code was: awk 'END { for (R in rec) { n = split(rec, t, "/") if (n > 1) dup = dup ?... (5 Replies)
Discussion started by: bibb
5 Replies

7. UNIX for Dummies Questions & Answers

Filter lines common in two files

Thanks everyone. I got that problem solved. I require one more help here. (Yes, UNIX definitely seems to be fun and useful, and I WILL eventually learn it for myself. But I am now on a different project and don't really have time to go through all the basics. So, I will really appreciate some... (6 Replies)
Discussion started by: latsyrc
6 Replies

8. Shell Programming and Scripting

Shell Script to find common lines and replace next line

I want to find common line in two files and replace the next line of first file with the next line of second file. (sed,awk,perl,bash any solution is welcomed ) Case Ignored. Multiple Occurrence of same line. File 1: hgacdavd sndm,ACNMSDC msgid "Rome" msgstr "" kgcksdcgfkdsb... (4 Replies)
Discussion started by: madira
4 Replies

9. Shell Programming and Scripting

Find common lines with one file and with all of the files in another folder

Hi! I would like to comm -12 with one file and with all of the files in another folder that has a 100 files or more (that file is not in that folder) to find common text lines. I would like to have each case that they have common lines to be written to a different output file and the names of the... (6 Replies)
Discussion started by: Eve
6 Replies

10. Shell Programming and Scripting

Find common lines between all of the files in one folder

Could it be possible to find common lines between all of the files in one folder? Just like comm -12 . So all of the files two at a time. I would like all of the outcomes to be written to a different files, and the file names could be simply numbers - 1 , 2 , 3 etc. All of the file names contain... (19 Replies)
Discussion started by: Eve
19 Replies

LEARN ABOUT OPENSOLARIS

noecho

echo(3XCURSES)						  X/Open Curses Library Functions					    echo(3XCURSES)

NAME

       echo, noecho - enable/disable terminal echo

SYNOPSIS

       cc [ flag... ] file... -I /usr/xpg4/include  -L	/usr/xpg4/lib 
	-R  /usr/xpg4/lib  -lcurses  [ library... ]

       c89 [ flag... ] file... -lcurses [ library... ]

       #include <curses.h>

       int echo(void);

       int noecho(void);

DESCRIPTION

       The  echo()  function enables Echo mode for the current screen. The noecho() function disables Echo mode for the current screen. Initially,
       curses software echo mode is enabled and hardware echo mode of the tty driver is disabled. The echo() and noecho() functions control  soft-
       ware echo only. Hardware echo must remain disabled for the duration of the application, else the behavior is undefined.

RETURN VALUES

       Upon successful completion, these functions return OK. Otherwise, they return ERR.

ERRORS

       No errors are defined.

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Interface Stability	     |Standard			   |
       +-----------------------------+-----------------------------+
       |MT-Level		     |Unsafe			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       getch(3XCURSES), getstr(3XCURSES), initscr(3XCURSES), libcurses(3XCURSES), scanw(3XCURSES), attributes(5), standards(5)

SunOS 5.11							    5 Jun 2002							    echo(3XCURSES)
All times are GMT -4. The time now is 04:00 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy