remove duplicated lines without sort

lalelle · #1 (**permalink**) 08-20-2007

Hi
Just wondering whether or not I can remove duplicated lines without sort

For example, I use the command who, which shows users who are logging on. In some cases, it shows duplicated lines of users who are logging on more than one terminal.

Normally, I would do

who | cut -d" " -f1 | sort |uniq

But I realised that I do not want data to be sorted.

Is there any other ways to do it?

porter · #2 (**permalink**) 08-20-2007

Are the duplicated lines adjacent? If not which one do you remove?

summer_cherry · #3 (**permalink**) 08-20-2007

Try follow one, use awk array to remember the appearance of the line:

Code:

awk '{
 if (arr[$0]=="")
 arr[$0]=1
 }
 END{
 for (i in arr)
 print i
 }' filename

lalelle · #4 (**permalink**) 08-20-2007

I am new to awk or you can say UNIX. Honestly, I do not understand awk at the moment.
I will try to study it.

Thank you for your reply anyway.

Duplicated lines are not adjacent and I can remove any of them.

Cheers,

radoulov · #5 (**permalink**) 08-21-2007

Code:

$ who
oracle     pts/1        Aug 21 04:57    (...)
oracle     pts/4        Aug 21 04:57    (...)
$ who|awk '!x[$1]++'
oracle     pts/1        Aug 21 04:57    (...)

If you want only the first field to be displayed:

Code:

who|awk '!x[$1]++{print $1}'

Use nawk on Solaris.

bigearsbilly · #6 (**permalink**) 08-21-2007

Code:

perl -ne '$H{$_}++ or print'

remove duplicate without sorting

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Help removing lines with duplicated columns	yahyaaa	Shell Programming and Scripting	14	05-17-2008 07:33 AM
Remove lines, Sorted with Time based columns using AWK & SORT	karthikn7974	Shell Programming and Scripting	1	05-09-2008 11:04 PM
How to remove duplicate records with out sort	svenkatareddy	SUN Solaris	2	02-28-2008 08:38 AM
remove duplicated columns	kamel.seg	Shell Programming and Scripting	6	02-21-2008 07:36 AM
remove duplicated xml record in a file under unix	happyv	Shell Programming and Scripting	8	09-20-2006 01:36 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by