Hi,

How can i delete the second and subsequent occurrence of a particular string from a file ?

eg) test.txt
cattle
bat
battle
mat
matter
cattle
cattle

my output file should be

cattle
bat
battle
mat
matter

I am new to unix and your advice is greatly appreciated.

Thanks in advance,
gops

Looks like you want to remove the duplicate entries.

Removing duplicates

Hi Vino,

Thanks for your prompt response.

Here i want to pass a string and only duplicate entry of that string should be deleted rather than all duplicates.

In the extract file which we have, lots of duplicates are available and we dont want to remove everything except the string which we specify explicitely.

Thanks in advance.

cheers,
gops

Hi Jim,

I have a problem executing the awk script which you have recommended.

I am using Korn shell. Do i need to change some thing to accommodate that.
Herewith i am attaching my file

test.txt
cat
cattle
bat
battle
mat
matter
fat
fatter
cattle
cattle

I am bit confused about the script. When i execute it, I am getting following error message.

awk: syntax error near line 1
awk: bailing out near line 1

Let me know what i should assign for $0 and $1.
Thanks in advance.
Since I am a novice, I have several queries from my end.

cheers,
gops

$1 is the string whose duplicates will be removed. Either you pass the string to that script or replace $0 by the string in that script.

Awk process the text file line by line. For each line read is placed in $0.

Could you use 'uniq'?

Hi Thomas,

I want to remove duplicates which are specified explicitely rather than all duplicates.
Thanks Thomas.

Hi Anbu,

I am getting the same error message. Can you take up my example and change the script accordingly.
Thanks in advance.

gops

Hi Anbu,

I was off for few days and hence the delay in replying.
I tried your stuff but I am getting this error
awk -v check_val="$1" '{
if( $0==check_val) { key[$0]++ }
if(key[$0] < 2 ) print $0
}' inputfile > outputfile

awk: syntax error near line 1
awk: bailing out near line 1

I created a script test and fired it like ksh test.ksh cattle

I am using korn shell. Do i need to make some changes for that ?

Thanks in advance.

cheers,
gops

Hi Guys,

It is working fine when i supstitute nawk with awk command. Anbu, thanks a lot.

When i pass the exact string that line gets deleted in the outputfile.

But i want to pass a part of the string and i want whole string to be deleted.

for eg

inputfile

cattle
battle
cat
mattle
cattle

my outputfile should be
cattle
battle
mattle

here i want to check for cat%. I dont know how to do that using nawk.
Please help me out on this.

Best Regards,
gops

umm..how abt this?

awk '/$1/' inputfile>temp

awk '{for ( $0 in `cat temp` )
key[$0]++
if(key[$0] < 2 ) print $0
}' inputfile > outputfile


i didnt try this out, but i hope the logic is clear. i searched for the pattern specified by $1 and stored all the matches in temp. then instead of a direct comparison with 'if', i matched $0 with all the values in temp using a for loop. rest of the logic is the same. pardon any syntactical errors.

Hi npolayan,

I can able to create the temp file with appropriate values to look at but when i executed the awk statement
awk '{ for ($0 in `cat temp`)
key[$0]++
if(key[$0] < 2 ) print $0
}' inputfile > outputfile

I am getting error messages

illegal statement at source line 1
illegal statement at source line 4

Let me know if you have the answer.

Thanks in advance.

cheers,
gops