UNIX for Dummies Questions & Answers

Hi,

How can i delete the second and subsequent occurrence of a particular string from a file ?

eg) test.txt
cattle
bat
battle
mat
matter
cattle
cattle

my output file should be

cattle
bat
battle
mat
matter

I am new to unix and your advice is greatly appreciated.

Thanks in advance,
gops

Looks like you want to remove the duplicate entries.

Removing duplicates

Hi Vino,

Thanks for your prompt response.

Here i want to pass a string and only duplicate entry of that string should be deleted rather than all duplicates.

In the extract file which we have, lots of duplicates are available and we dont want to remove everything except the string which we specify explicitely.

Thanks in advance.

cheers,
gops

Code:

#!/bin/ksh

awk -v check_val="$1" '{
        if( $0==check_val) { key[$0]++ }
        if(key[$0} < 2 ) print $0
       }' inputfile > outputfile

Hi Jim,

I have a problem executing the awk script which you have recommended.

I am using Korn shell. Do i need to change some thing to accommodate that.
Herewith i am attaching my file

test.txt
cat
cattle
bat
battle
mat
matter
fat
fatter
cattle
cattle

I am bit confused about the script. When i execute it, I am getting following error message.

awk: syntax error near line 1
awk: bailing out near line 1

Let me know what i should assign for $0 and $1.
Thanks in advance.
Since I am a novice, I have several queries from my end.

cheers,
gops

$1 is the string whose duplicates will be removed. Either you pass the string to that script or replace $0 by the string in that script.

Awk process the text file line by line. For each line read is placed in $0.

Could you use 'uniq'?

Code:

uniq -u test.txt
cat
cattle
bat
battle
mat
matter
fat
fatter