Remove duplicate


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove duplicate
# 8  
Old 05-24-2013
Solaris /bin/awk -> oawk does not allow a ! operator on a number.
Code:
awk -F\| '0==L[$1]++'

# 9  
Old 06-07-2013
Hi,
My .psv file is getting bigger as more columns are being added. I remove duplicates based on the last column number and currently i know the position of this which is 176 . The column name is 'auditid'. Is there a way i can find the column number of this field and assign it to the array ? .


nawk -F\| '!L[$176]++' file
# 10  
Old 06-07-2013
Assuming the column name is a field in row 1, try
Code:
awk -F\|  'NR==1 {for (i=1; i<=NF; i++) if ($i==COLNAME){COL=i; break}} !L[$COL]++' COLNAME="auditid" file

You can use the column name as a constant for the comparison, or, like here, pass it i a variable.
This User Gave Thanks to RudiC For This Post:
# 11  
Old 06-07-2013
Quote:
Originally Posted by RudiC
Try
Code:
$ awk '!L[$1]++' file
15277105||Common Stick|ESHR||Common Stock|CYRO AB
16111278||Common Stick|ESHR||Common Stock|STANDARD REGISTER CO
39693766||Common Stick|ESHR||Common Stock|HS AG

You may need to redefine the field separator (man awk) if you want to use this on other files.
Just wondering... what is the logic behind using L[$1]++?
# 12  
Old 06-07-2013
L is just an arbitrary Name for an array - call it what you like, Joe, Mimi, or L (short for logical). The $1 (first field of each respective row/line) is the index into that array, and that indexed element is incremented by ++. Any value except 0 (or empty, which is equivalent) will make the reference true, its inversion (by !) false. As the default action is print, the entire command reads: Get the array element for index $1. If it does not exist (=first occurrence of this index) invert to true, print. If it does (and has a value) , invert to false, don't print. Increment it for later print suppression.
# 13  
Old 06-07-2013
Thank you, RudiC for the explanation. It was really helpful Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR (10 Replies)
Discussion started by: tara123
10 Replies

3. UNIX for Dummies Questions & Answers

Remove duplicate

Hi, How can I replace || with space and then remove duplicate from following text? T111||T222||T444||T222||T555 Thanks in advance (10 Replies)
Discussion started by: tinku981
10 Replies

4. Shell Programming and Scripting

How to remove duplicate ID's?

HI I have file contains 1000'f of duplicate id's with (upper and lower first character) as below i/p: a411532A411532a508661A508661c411532C411532 Requirement: But i need to ignore lowercase id's and need only below id's o/p: A411532 A508661 C411532 (9 Replies)
Discussion started by: buzzme
9 Replies

5. Shell Programming and Scripting

Help with remove duplicate content

Input file data_1 10 US data_1 2 US data_1 5 UK data_2 20 ENGLAND data_2 12 KOREA data_3 4 CHINA . . data_60 123 US data_60 23 UK data_60 45 US Desired output file data_1 10 US data_1 5 UK data_2 20 ENGLAND data_2 12 KOREA (2 Replies)
Discussion started by: perl_beginner
2 Replies

6. Shell Programming and Scripting

remove duplicate

Hi, I am tryung to use shell or perl to remove duplicate characters for example , if I have " I love google" it will become I love ggle" or even "I loveggle" if removing duplicate white space Thanks CC (6 Replies)
Discussion started by: ccp
6 Replies

7. UNIX for Dummies Questions & Answers

Remove duplicate in array

Hi, I have a list of numbers stored in an array as below. 5 7 10 30 30 40 50 Please advise how could I remove the duplicate value in the array ? Thanks in advance. (5 Replies)
Discussion started by: Rock
5 Replies

8. Shell Programming and Scripting

Remove duplicate

Hi all, I have a text file fileA.txt DXRV|02/28/2006 11:36:49.049|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:37:06.404|SAC||||CDxAcct=6070970034 DXRV|02/28/2006 11:37:25.740|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:38:32.633|SAC||||CDxAcct=6070970034 DXRV|02/28/2006... (2 Replies)
Discussion started by: sabercats
2 Replies

9. Shell Programming and Scripting

Remove duplicate ???

Hi all, I have a out.log file CARR|02/26/2006 10:58:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:11:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:18:30.107|CDxAcct=7659579782 CARR|02/26/2006 11:28:30.107|CDxAcct=9534922327 CARR|02/26/2006 11:38:30.107|CDxAcct=9534922327 CARR|02/26/2006... (3 Replies)
Discussion started by: sabercats
3 Replies

10. Shell Programming and Scripting

remove duplicate

i have a text its contain many record, but its written in one line, i want to remove from that line the duplicate record, not record have fixed width ex: width = 4 inputfile test.txt =abc cdf abc abc cdf fgh fgh abc abc i want the outputfile =abc cdf fgh only those records can any one help... (4 Replies)
Discussion started by: kazanoova2
4 Replies
Login or Register to Ask a Question