![]() |
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.
|
|
google unix.com
|
|||||||
| Forums | Register | Forum Rules | Links | Albums | FAQ | Members List | Calendar | Search | Today's Posts | Mark Forums Read |
| Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here. |
More UNIX and Linux Forum Topics You Might Find Helpful
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| How to remove duplicate records with out sort | svenkatareddy | Shell Programming and Scripting | 19 | 06-11-2008 02:10 PM |
| How to remove duplicate records with out sort | svenkatareddy | SUN Solaris | 2 | 02-28-2008 08:38 AM |
| Duplicate records from oracle to text file. | shilendrajadon | UNIX for Advanced & Expert Users | 1 | 01-10-2008 11:21 AM |
| Delete Duplicate records from a tilde delimited file | irshadm | Shell Programming and Scripting | 5 | 12-06-2007 05:36 AM |
| Remove Duplicate lines from File | Nysif Steve | UNIX for Dummies Questions & Answers | 18 | 09-09-2007 08:57 AM |
![]() |
|
|
LinkBack | Thread Tools | Search this Thread | Rate Thread | Display Modes |
|
||||
|
Remove all instances of duplicate records from the file
Hi experts,
I am new to scripting. I have a requirement as below. File1: A|123|NAME1 A|123|NAME2 B|123|NAME3 File2: C|123|NAME4 C|123|NAME5 D|123|NAME6 1) I have 2 merge both the files. 2) need to do a sort ( key fields are first and second field) 3) remove all the instances of duplicate records from the merged file and write write all these duplicate instances into one file. 4) rest of the records which are unique in the original source files, needs to be written into another file outfiles: file3: A|123|NAME1 A|123|NAME2 C|123|NAME4 C|123|NAME5 File4: B|123|NAME3 D|123|NAME6 Please help me with the solution as I am in real urgent. Appreciate your help. Thank you |
|
||||
|
Quote:
SO if all data are unique all the records should go to File4 ..Is n't it? Explain more clearly, so that you ll get a quick reply from this forum. Beleive me here in this forum really brilliant and experts here to help you out at any time.. excluding me .. ![]() Cheers user_prady |
|
|||||
|
Another sort/Awk solution
(if your files are not already sorted as the samples you posted): Code:
sort -t\| -k1,2 file1 file2|awk '{
x[$1,$2]++
y[NR] = $0
} END {
for (i = 1; i <= NR; i++)
print y[i] > ((x[substr(y[i],1,5)] > 1) ? "file3" : "file4")
}' SUBSEP="|" FS="|"
P.S. For variable column width: you should not use substr, but split for example: Code:
sort -t\| -k1,2 file1 file2|awk '{
x[$1,$2]++
y[NR] = $0
} END {
for (i = 1; i <= NR; i++)
{
tmp = y[i]
split(tmp,z)
print tmp > ((x[z[1],z[2]] > 1) ? "file3" : "file4")
}
}' SUBSEP="|" FS="|"
Last edited by radoulov; 12-12-2007 at 08:16 AM.. |
| Sponsored Links | ||
|
|