duplicate line in a text file

nixguy · #1 (**permalink**) 04-25-2008

i would like to scan file in for duplicate lines, and print the duplicates to another file,
oh and it has to be case insensitive.

example

line1
line2
line2
line3
line4
line4

outputfile:
line2
line4

any ideas

era · #2 (**permalink**) 04-25-2008

Code:

perl -ne '$l = lc(); print if $m{$l}++; ' file

lc returns the specified string in lowercase; in this case, with no parameter, it defaults to the current input line. $m{$l} is the count of number of times we have seen $l; if it's nonzero, it's a duplicate, so we print it.

nixguy · #3 (**permalink**) 04-25-2008

Quote:

Originally Posted by era

Code:

perl -ne '$l = lc(); print if $m{$l}++; ' file

lc returns the specified string in lowercase; in this case, with no parameter, it defaults to the current input line. $m{$l} is the count of number of times we have seen $l; if it's nonzero, it's a duplicate, so we print it.

thanks it's worked for one file, but i forgot to add that if i need it to run on a directory and make sure to do in on a file only and not subdirectory

what modifications would you do

era · #4 (**permalink**) 04-25-2008

Loop over files in a directory, and remove duplicates? Do you want to replace the files?

Code:

for f in directory/*; do
  test -d "$f" && continue   # skip if it's a subdirectory
  perl -ne '$l = lc(); print if $m{$l}++; ' "$f" >"$f.tmp"
  mv "$f.tmp" "$f"
done

nixguy · #5 (**permalink**) 04-25-2008

Thank you Era, it worked,

summer_cherry · #6 (**permalink**) 04-29-2008

Code:

awk '{
  a[$0]++
}
END{
 for (i in a)
 if (a[i]=2)
 print i
}' filename

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
concatenate all duplicate line in a file.	vaskarbasak	Shell Programming and Scripting	30	08-29-2008 05:31 AM
reading text file line by line	MizzGail	Shell Programming and Scripting	6	04-14-2008 06:58 AM
Eleminating Duplicate IPs from a text file	coolkid	Shell Programming and Scripting	4	01-24-2008 07:22 PM
Duplicate records from oracle to text file.	shilendrajadon	UNIX for Advanced & Expert Users	1	01-10-2008 11:21 AM
paging a text file line by line?	tomapam	UNIX for Dummies Questions & Answers	2	10-07-2002 09:30 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread: