How to remove duplicate lines of a record without changing the order

abhi.roy03 · #1 (**permalink**) 08-02-2008

Hi all,
I have to remove duplicate lines in a file without chainging the order.for eg if i have a record

pqr
def
abc
lmn
pqr
abc
mkh
hgf

the output should be
pqr
def
abc
lmn
mkh
hgf

Plz help me.It is urgent

Abhishek

radoulov · #2 (**permalink**) 08-02-2008

Is this a homework?
Try searching the forums.

redoubtable · #3 (**permalink**) 08-02-2008

Code:

#!/usr/bin/perl -w

while (<STDIN>)
{
        push (@lines, $_);
}

print "-\n";

foreach my $i (@lines)
{
        if (scalar (grep { /$i/ } @lines) == 1)
        {
                print $i;
        }
}

Usage:

Code:

Tsunami repeated_lines # perl repeat.pl 
pqr
def
abc
lmn
pqr
abc
mkh
hgf
-
def
lmn
mkh
hgf
Tsunami repeated_lines #

To stop input, just hit Ctrl-D and the script will give you all the non-repeated strings in the input order.

You can also do something like:

Code:

Tsunami repeated_lines # cat lines |perl repeat.pl 
-
def
lmn
mkh
hgf
Tsunami repeated_lines # cat lines 
pqr
def
abc
lmn
pqr
abc
mkh
hgf
Tsunami repeated_lines #

fpmurphy · #4 (**permalink**) 08-02-2008

Using awk ...

Code:

$ awk '! a[$0]++' file
pqr
def
abc
lmn
mkh
hgf
$

radoulov · #5 (**permalink**) 08-02-2008

As I said,
you just need to search the forums to find the following solutions:

Code:

awk '!_[$0]++' input

Code:

perl -ne'print unless $_{$_}++' input

or:

Code:

perl -ne'$_{$_}++||print' input

era · #6 (**permalink**) 08-03-2008

If the requirement is to keep the last instead of the first occurrence, it's only marginally harder.

Code:

perl -ne '$n{$_} = $.; END { print sort { $n{$a} <=> $n{$b} } keys %n }'

If the last line lacks a newline, that will count as a unique line. It's not terribly hard to fix, but I didn't want to complicate the script.

radoulov · #7 (**permalink**) 08-03-2008

Quote:

If the requirement is to keep the last instead of the first occurrence
[...]

With GNU Awk:

Code:

awk 'END { for (k in r) t[sprintf("%10d", r[k])] = k
  n = asorti(t, _)
  while (++i <= n) print t[_[i]] }
{ r[$0] = NR }' filename

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Remove duplicate lines in log files	karthikn7974	Shell Programming and Scripting	4	03-21-2009 06:41 PM
Remove Duplicate lines from File	Nysif Steve	UNIX for Dummies Questions & Answers	18	09-09-2007 08:57 AM
how to remove duplicate lines	fredao	Shell Programming and Scripting	3	12-13-2006 12:51 PM
Remove Duplicate Lines in File	Teh Tiack Ein	Shell Programming and Scripting	5	01-12-2006 08:30 AM
reconstructing a record in a diffrent order	r1500	UNIX for Dummies Questions & Answers	1	10-16-2003 05:24 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread: