Shell Programming and Scripting

Code:

 
sed '$!N; /^\(.*\)\n\1$/!P; D'

i found this file which removes duplicates irrespective for sorted or unsorted file. keep first occurance and remove the further occurances.

can any1 explain how this is working..

i need to remove duplicates following file. duplicate criteria is not the complete line, but only first 3 parameters. ie (a,b,c) or (e,r,t)

Code:

 
a,b,c,d
a,b,c,s
e,r,t,y
a,b,c,a
a,b,c,e
e,r,t,y

i need an output like

Code:

 
a,b,c,d
e,r,t,s

taking the first occurance and ignoring the rest.

please help. i am getting nowhere from it.

Do you mean this?

Code:

awk '!++a[$1$2$3]' FS=","

Funny but the above didn't work for me I needed the following:

Code:

awk -F, '!a[$1,$2,$3]++'

Note: extra commas are to avoid false match between "aa,b,c" and "a,ab,c" or ",a,b" and "a,,b"

Code:

 
awk -F, '!a[$1,$2,$3]++' UnixEg.dat

i tried above, but got an error

Code:

a[$1,$2,$3]++': Event not found

guess could be beacause of ! . escaped by \

Code:

 
awk -F, '\!a[$1,$2,$3]++' UnixEg.dat

then got

Code:

awk: syntax error near line 1
awk: bailing out near line 1

am i missing something. trying to run in csh shell.

---------- Post updated at 10:48 PM ---------- Previous update was at 10:39 PM ----------

fixed it though.

changed to ksh shell
and ran this

Code:

 
nawk -F, '!a[$1,$2,$3]++' UnixEg.dat

works superbly.

thanks man.