Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Print only the duplicate line only with matching columns

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Print only the duplicate line only with matching columns
# 1  
Old 11-20-2015
Print only the duplicate line only with matching columns
Hi There,

I have an I/P which looks like --

Code:
1 2 3 4 5
1 2 3 4 6
4 7 8 9 9 
5 6 7 8 9

I would like O/P to be ---

Code:
1 2 3 4 5
1 2 3 4 6

So, printing only the consecutive lines where $1,$2,$3,$4 are matching.
Is there any command to do this or small awk script?

Thanks,
# 2  
Old 11-21-2015
Hello Indra2011,

Let's say we have following Input_file:
Code:
cat Input_file
1 2 3 4 5
1 2 3 4 6
4 7 8 9 9
5 6 7 8 9
2 3 4 5 6
2 3 4 5 9
3 4 5 9 7
3 4 5 6 8

Then as per your conditions provided in post, could you please try following code and let me know if that helps.
Code:
awk '{if($1 FS $2 FS $3 FS $4 == Y && NR%2==0){print Q ORS $0}};{Q=$0;Y=$1 FS $2 FS $3 FS $4}'  Input_file

Output will be as follows.
Code:
1 2 3 4 5
1 2 3 4 6
2 3 4 5 6
2 3 4 5 9

If this code doesn't match with your requirement and you have more conditions to your requirement, request you to please show a bit more sample Input_file with sample output too.

EDIT: Let's say you have more than 2 consecutive lines and you want to match all, then following may help.
Following is the Input_file:
Code:
1 2 3 4 5
1 2 3 4 6
1 2 3 4 6
1 2 3 4 6
4 7 8 9 9
5 6 7 8 9
2 3 4 5 6
2 3 4 5 6
2 3 4 5 6
2 3 4 5 6
2 3 4 5 9
3 4 5 9 7
3 4 5 6 8
2 3 4 5 6
3 4 5 6 8

Then following is the code for above Input_file:
Code:
awk '{if($1 FS $2 FS $3 FS $4 != Y){i=""};{if($1 FS $2 FS $3 FS $4 == Y){if(i>=1){print $0} else {print Q ORS $0;i++};}}};{Q=$0;Y=$1 FS $2 FS $3 FS $4}' Input_file

Output will be as follows.
Code:
1 2 3 4 5
1 2 3 4 6
1 2 3 4 6
1 2 3 4 6
2 3 4 5 6
2 3 4 5 6
2 3 4 5 6
2 3 4 5 6
2 3 4 5 9

Thanks,
R. Singh
Last edited by RavinderSingh13; 11-21-2015 at 02:33 AM.. Reason: Changed Input_file a bit. Added one more condition with solution too.
This User Gave Thanks to RavinderSingh13 For This Post:
Indra2011
# 3  
Old 11-21-2015
Untested
Code:
awk '{if ((k=$1 FS $2 FS $3 FS $4) == pk) {print p0 $0; p0=""} else {p0=$0 ORS; pk=k}}'  Input_file

Last edited by MadeInGermany; 11-21-2015 at 03:07 AM.. Reason: pos of (
This User Gave Thanks to MadeInGermany For This Post:
Indra2011
# 4  
Old 11-21-2015
Many many thanks Ravinder and Made in Germany :-)
# 5  
Old 11-23-2015
Dear Gurus,
Is there also a way that we can print only the non-uniq records?
Like IP file is --
Code:
1 2 3 4 5
1 2 3 4 6
4 7 8 9 9
5 6 7 8 9
2 3 4 5 6
2 3 4 5 9
3 4 5 9 7
3 4 5 6 8

And we would liek to print --
Code:
4 7 8 9 9
5 6 7 8 9

thanks,
Last edited by Indra2011; 11-23-2015 at 02:25 PM..
# 6  
Old 11-23-2015
Hello Indra,

Following may help you in same. Again considering that you are comparing only consecutive 2 lines each time.
Code:
wk '{if(Y!=$1 FS $2 FS $3 FS $4 && NR%2==0){print Q ORS $0;}};{Y=$1 FS $2 FS $3 FS $4;Q=$0}' Input_file

Output will be as follows.
Code:
4 7 8 9 9
5 6 7 8 9
3 4 5 9 7
3 4 5 6 8

Also not sure why you haven't mentioned last 2 lines of above output in your required sample output as per your condition their first four columns too are not matching so they should come too, let me know if you have any queries.


Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
Indra2011
# 7  
Old 11-23-2015
Many many thanks Ravinder.
you are absolutely correct about that i missed the last two lines of the O/P and they should be there.

Will these code of your's be modified to work if we have more than 2 consecutive lines.
I/P file -
Code:
1 2 3 4 5
1 2 3 4 6
1 2 3 4 6
1 2 3 4 6
4 7 8 9 9
5 6 7 8 9
2 3 4 5 6
2 3 4 5 6
2 3 4 5 6
2 3 4 5 6
2 3 4 5 9
3 4 5 9 7
3 4 5 6 8
2 3 4 5 6
3 4 5 6 8

and the Op as follows
Code:
4 7 8 9 9
5 6 7 8 9
3 4 5 9 7
3 4 5 6 8
2 3 4 5 6
3 4 5 6 8

Thanks a lot.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to copy previous line matching a particular columns

Hello Help, 2356798 7689867 999 000 123678 20385907 9797 666 17978975 87468976 968978 98798 I am trying to have out put which actually look for the third column value of 9797 and then it insert line there after with first, second column value exactly as the previous line and replace the third... (3 Replies)
Discussion started by: Indra2011
3 Replies

2. Shell Programming and Scripting

Compare file1 for matching line in file2 and print the difference in matching lines

Hello, I have two files file 1 and file 2 each having result of a query on certain database tables and need to compare for Col1 in file1 with Col3 in file2, compare Col2 with Col4 and output the value of Col1 from File1 which is a) not present in Col3 of File2 b) value of Col2 is different from... (2 Replies)
Discussion started by: RasB15
2 Replies

3. Shell Programming and Scripting

Matching pattern in two files and print the line

Hi, I want to match the pattern in file1 with file2 and print the value in file2 and paste in file1 file1: ISHO RT SR Major 96.46778 Drop Call Rate CS Critical 0.5072662 ISHO RT SR Major 97.754364... (3 Replies)
Discussion started by: khalil
3 Replies

4. Shell Programming and Scripting

Duplicate line removal matching some columns only

I'm looking to remove duplicate rows from a CSV file with a twist. The first row is a header. There are 31 columns. I want to remove duplicates when the first 29 rows are identical ignoring row 30 and 31 BUT the duplicate that is kept should have the shortest total character length in rows 30... (6 Replies)
Discussion started by: Michael Stora
6 Replies

5. Shell Programming and Scripting

Print columns matching to specific values

Hello Friends, I have a CDR file and i need to print out 2 columns with their field position which matches to some constant values, a part of input file CZ=1|CZA=1|DIAL=415483420001|EE=13|ESF=1|ET=|FF=0|9|MNC=99|MNP=9041|MTC=0|NID=2|NOA=international|ON=1| OutPut ... (3 Replies)
Discussion started by: EAGL€
3 Replies

6. Shell Programming and Scripting

print range of lines matching pattern and previous line

Hi all, on Solaris 10, I'd like to print a range of lines starting at pattern but also including the very first line before pattern. the following doesn't print the range starting at pattern and going down to the end of file: cat <my file> | sed -n -e '/<pattern>{x;p;}/' I need to include the... (1 Reply)
Discussion started by: siriche
1 Replies

7. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

8. Shell Programming and Scripting

compare two columns of different files and print the matching second file..

Hi, I have two tab separated files; file1: S.No ddi fi cu o/l t+ t- 1 0.5 0.6 o 0.1 0.2 2 0.2 0.3 l 0.3 0.4 3 0.5 0.8 l 0.1 0.6 ... (5 Replies)
Discussion started by: vasanth.vadalur
5 Replies

9. Shell Programming and Scripting

using command line arguments as columns for pattern matching using awk

Hi, I wish to use a column, as inputted by a user from command line, for pattern matching. awk file: { if($1 ~ /^8/) { print $0> "temp2.csv" } } something like this, but i want '$1' to be any column as selected by the user from command line. ... (1 Reply)
Discussion started by: invinclible0009
1 Replies

10. UNIX for Dummies Questions & Answers

exclude columns with a matching line pattern

Hi , I have 5 columns total and am wanting to search lines in columns 3-5 and basically grep -v patterns that match 'BBB_0123' 'BVG_0895' 'BSD_0987' Does anyone know how to do this? I tried combining grep -v with grep -e but, it didn't work. Thanks! (5 Replies)
Discussion started by: greptastic
5 Replies
Login or Register to Ask a Question