filter the uniq record problem


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting filter the uniq record problem
# 1  
Old 05-12-2009
filter the uniq record problem

Anyone can help for filter the uniq record for below example? Thank you very much

Input file
20090503011111|test|abc
20090503011112|tet1|abc|def
20090503011112|test1|bcd|def
20090503011131|abc|abc
20090503011131|bbc|bcd
20090503011152|bcd|abc
20090503011151|abc|abc
20090503011151|abc|bcd
20090503011162|bcd|abc

Expect outupt
20090503011111|test|abc
20090503011152|bcd|abc
20090503011162|bcd|abc
# 2  
Old 05-12-2009
Don't make sense.
All lines are unique.
If you sort by column, by the string of numbers, it's 6 unique lines.
What's the criteria?
# 3  
Old 05-12-2009
Quote:
Originally Posted by System Shock
Don't make sense.
All lines are unique.
If you sort by column, by the string of numbers, it's 6 unique lines.
What's the criteria?

The highlighted line below are treat as duplicate record.....

Input file
20090503011111|test|abc
20090503011112|tet1|abc|def
20090503011112|test1|bcd|def
20090503011131|abc|abc
20090503011131|bbc|bcd
20090503011152|bcd|abc
20090503011151|abc|abc
20090503011151|abc|bcd
20090503011162|bcd|abc

Expect outupt
20090503011111|test|abc
20090503011152|bcd|abc
20090503011162|bcd|abc
# 4  
Old 05-13-2009
... so, if a string of numbers appears in 2 lines, both lines get deleted?
Or are you saying that all of the numbers in red are duplicates of each other?
# 5  
Old 05-13-2009
Quote:
Originally Posted by System Shock
... so, if a string of numbers appears in 2 lines, both lines get deleted?
Or are you saying that all of the numbers in red are duplicates of each other?
yes, both lines get deleted.
# 6  
Old 05-13-2009
Try this code

Code:
 
grep "`awk -F"|" '{print $1}' $fileName|sort|uniq -c|awk '$1 ~1 {print $2}'`" $fileName

I test it on the RedHat Linux,it works.
# 7  
Old 05-13-2009
With awk (use gawk, nawk or /usr/xpg4/bin/awk on Solaris):

Code:
awk -F\| 'END { 
  for (K in k)
    if (k[K] == 1) print r[K]
  }    
{ k[$1]++; r[$1] = $0 }
' infile

You may loose the original order, to preserve it index the second array by NR and then loop over in order.

With uniq from the GNU coreutils (i.e. on GNU Linux):

(assuming your input is already ordered and a fixed format like the one you posted)

Code:
uniq -dw14 infile

If the input data is not sorted:

Code:
sort infile|uniq -dw14

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

CSV File:Filter duplicate records from column1 & another column having unique record

Hi Experts, I have csv file with 30, 40 columns Pasting just 2 column for problem description. Need to print error if below combination is not present in file check for column-1 (DocumentNumber) and filter columns where value in DocumentNumber field is same. For all such rows, the field... (7 Replies)
Discussion started by: as7951
7 Replies

2. Shell Programming and Scripting

Filter uniq field values (non-substring)

Hello, I want to filter column based on string value. All substring matches are filtered out and only unique master strings are picked up. infile: 1 abcd 2 abc 3 abcd 4 cdef 5 efgh 6 efgh 7 efx 8 fgh Outfile: 1 abcd 4 cdef 5 efgh 7 efxI have tried awk '!a++; match(a, $2)>0'... (32 Replies)
Discussion started by: yifangt
32 Replies

3. Shell Programming and Scripting

Delete record filter by column

Dear friend, I have a file 2 files with column wise FILE_A ------------------------------ x,1,@ y,3,$ x,5,% FILE_B -------------------- x,1,@ i like to delete the all lines in FILE_A ,if first column available in FILE_B. output (in FILE_A) y,3,$ x,5,% (10 Replies)
Discussion started by: Jewel
10 Replies

4. Shell Programming and Scripting

Fetching record based on Uniq Key from huge file.

Hi i want to fetch 100k record from a file which is looking like as below. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ... (17 Replies)
Discussion started by: lathigara
17 Replies

5. Shell Programming and Scripting

awk-filter record by another file

I have file1 3049 3138 4672 22631 45324 112382 121240 125470 130289 186128 193996 194002 202776 228002 253221 273523 284601 284605 641858 (8 Replies)
Discussion started by: biomed
8 Replies

6. Shell Programming and Scripting

sort & uniq on specific fields problem

Hello; I have the output data set from: egrep -i 'warning| error| fail' /var/adm/syslog/syslog.log Jan 31 12:02:18 fidsrv vmunix: LVM: WARNING: VG 128 0x001000: LV 5: Some I/O requests to this LV are waiting Jan 31 12:02:23 fidsrv vmunix: Asynchronous write failed on LUN (dev=0x100000f)... (3 Replies)
Discussion started by: delphys
3 Replies

7. Shell Programming and Scripting

Keep the last uniq record only

Hi folks, Below is the content of a file 'tmp.dat', and I want to keep the uniq record (key by first column). However, the uniq record should be the last record. 302293022|2|744124889|744124889 302293022|3|744124889|744124889 302293022|4|744124889|744124889 302293022|5|744124889|744124889... (4 Replies)
Discussion started by: ChicagoBlues
4 Replies

8. Shell Programming and Scripting

Filter record from a file

Reposting since I didnt not get any reply. I have a problem while filtering records from a file. Can somebody help please? For eg: Consider the below files Record file: 0003@00000000000190@20100401@201004012010040120100401@003@... (1 Reply)
Discussion started by: gpaulose
1 Replies

9. Shell Programming and Scripting

filter record from a file reading another file

Hi, I want to filter record from a file if the records in the second column matches the data in another file. I tried the below awk command but it filters the records in the filter file. I want the opposite, to include only the records in the filter file. I tried this: awk -F'|'... (8 Replies)
Discussion started by: gpaulose
8 Replies

10. Shell Programming and Scripting

Help in writing a KSH script to filter the latest record?

Hi All, I have a text file with the folowing content. BANGALORE|1417|2010-02-04 08:41:04.174|dob|xxx BANGALORE|1416|2010-02-04 08:23:19.566|dob|yyy BANGALORE|1415|2010-02-04 08:20:14.497|dob|aaa BANGALORE|1414|2010-02-04 08:19:40.065|dob|vvv BANGALORE|1413|2010-02-04... (4 Replies)
Discussion started by: Karpak
4 Replies
Login or Register to Ask a Question