Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Print only the duplicate line only with matching columns

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Print only the duplicate line only with matching columns
# 8  
Old 11-23-2015
An awk approach by joining the input file:-
Code:
awk '
        NR == FNR {
                ++A[$1,$2,$3,$4]
                next
        }
        A[$1,$2,$3,$4] == 1
' file file

This User Gave Thanks to Yoda For This Post:
Indra2011
# 9  
Old 11-23-2015
For the fun of it:
Code:
cut -d' ' -f1-4 file | uniq -d | grep -vf- file | uniq -u
4 7 8 9 9
5 6 7 8 9
3 4 5 9 7

This User Gave Thanks to RudiC For This Post:
Indra2011
# 10  
Old 11-23-2015
Million thanks to Ravinder, Yoda and RudiC.

Hi Yoda ---
Your script is not running for a long file of say 100000 records.

Hi RudiC,
Your script is running fine but it's taking a long time.

I really appreciate you guys skill in scripting, you can create almost anuthing.

Thanks.
# 11  
Old 11-24-2015
Would this be faster?
Code:
sort file | awk '{key = $1 FS $2 FS $3 FS $4} LK != key {if (CNT == 1 && NR > 1) print last; CNT=0} {LK = key; CNT++; last = $0} END {if (CNT == 1) print}'
3 4 5 9 7
4 7 8 9 9
5 6 7 8 9

This User Gave Thanks to RudiC For This Post:
Indra2011
# 12  
Old 11-24-2015
How fast is this?
Code:
awk '{if ((k=$1 FS $2 FS $3 FS $4) == pk) {f=0} else {if (f==1) {print p0} else {f=1} p0=$0; pk=k}} END {if (f==1) print p0}' Input_file

This User Gave Thanks to MadeInGermany For This Post:
Indra2011
# 13  
Old 11-24-2015
Many many thanks you all the xperts there.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to copy previous line matching a particular columns

Hello Help, 2356798 7689867 999 000 123678 20385907 9797 666 17978975 87468976 968978 98798 I am trying to have out put which actually look for the third column value of 9797 and then it insert line there after with first, second column value exactly as the previous line and replace the third... (3 Replies)
Discussion started by: Indra2011
3 Replies

2. Shell Programming and Scripting

Compare file1 for matching line in file2 and print the difference in matching lines

Hello, I have two files file 1 and file 2 each having result of a query on certain database tables and need to compare for Col1 in file1 with Col3 in file2, compare Col2 with Col4 and output the value of Col1 from File1 which is a) not present in Col3 of File2 b) value of Col2 is different from... (2 Replies)
Discussion started by: RasB15
2 Replies

3. Shell Programming and Scripting

Matching pattern in two files and print the line

Hi, I want to match the pattern in file1 with file2 and print the value in file2 and paste in file1 file1: ISHO RT SR Major 96.46778 Drop Call Rate CS Critical 0.5072662 ISHO RT SR Major 97.754364... (3 Replies)
Discussion started by: khalil
3 Replies

4. Shell Programming and Scripting

Duplicate line removal matching some columns only

I'm looking to remove duplicate rows from a CSV file with a twist. The first row is a header. There are 31 columns. I want to remove duplicates when the first 29 rows are identical ignoring row 30 and 31 BUT the duplicate that is kept should have the shortest total character length in rows 30... (6 Replies)
Discussion started by: Michael Stora
6 Replies

5. Shell Programming and Scripting

Print columns matching to specific values

Hello Friends, I have a CDR file and i need to print out 2 columns with their field position which matches to some constant values, a part of input file CZ=1|CZA=1|DIAL=415483420001|EE=13|ESF=1|ET=|FF=0|9|MNC=99|MNP=9041|MTC=0|NID=2|NOA=international|ON=1| OutPut ... (3 Replies)
Discussion started by: EAGL€
3 Replies

6. Shell Programming and Scripting

print range of lines matching pattern and previous line

Hi all, on Solaris 10, I'd like to print a range of lines starting at pattern but also including the very first line before pattern. the following doesn't print the range starting at pattern and going down to the end of file: cat <my file> | sed -n -e '/<pattern>{x;p;}/' I need to include the... (1 Reply)
Discussion started by: siriche
1 Replies

7. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

8. Shell Programming and Scripting

compare two columns of different files and print the matching second file..

Hi, I have two tab separated files; file1: S.No ddi fi cu o/l t+ t- 1 0.5 0.6 o 0.1 0.2 2 0.2 0.3 l 0.3 0.4 3 0.5 0.8 l 0.1 0.6 ... (5 Replies)
Discussion started by: vasanth.vadalur
5 Replies

9. Shell Programming and Scripting

using command line arguments as columns for pattern matching using awk

Hi, I wish to use a column, as inputted by a user from command line, for pattern matching. awk file: { if($1 ~ /^8/) { print $0> "temp2.csv" } } something like this, but i want '$1' to be any column as selected by the user from command line. ... (1 Reply)
Discussion started by: invinclible0009
1 Replies

10. UNIX for Dummies Questions & Answers

exclude columns with a matching line pattern

Hi , I have 5 columns total and am wanting to search lines in columns 3-5 and basically grep -v patterns that match 'BBB_0123' 'BVG_0895' 'BSD_0987' Does anyone know how to do this? I tried combining grep -v with grep -e but, it didn't work. Thanks! (5 Replies)
Discussion started by: greptastic
5 Replies
Login or Register to Ask a Question