Identify duplicate words in a line using command

srinivasan_85 · #1 (**permalink**) 04-27-2007

Hi,
Let me explain the problem clearly:
Let the entries in my file be:

Code:

lion,tiger,bear
apple,mango,orange,apple,grape
unix,windows,solaris,windows,linux
red,blue,green,yellow
orange,maroon,pink,violet,orange,pink

Can we detect the lines in which one of the words(separated by field separator) occurs more than once, using a command (or command pipe)?
In this case, the command should detect the lines 2,3,5.

I accomplished it using a perl script (cited below), although i wonder whether this could be done through a command (the difficulty is that the no. of columns is not constant).

Perl program that I used:

Code:

$fname=<STDIN>;
chomp $fname;
open(file,"<$fname");
$found_dups=0;

for $line(<file>)
{
  chomp $line;
  @arr=split(/,/,$line);
  for($i=1;$i<=$#arr;$i++)
  {
     for($j=$i+1;$j<=$#arr;$j++)
     {
        if($arr[$i] eq $arr[$j])
        {
           print "tid $arr[0]\n";
           $found_dups++;
        }
     }
  }
}
print "Found $found_dups duplicates\n";

Thanks,
Srini

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
count no of words in a line	satish@123	Shell Programming and Scripting	7	05-21-2008 03:59 AM
Need to identify the line containing @ in between the line of a file	b.paramanatti	UNIX for Dummies Questions & Answers	4	11-04-2007 10:50 PM
seperating the words from a line??	skyineyes	Shell Programming and Scripting	3	06-26-2007 10:00 AM
removing line and duplicate line	ocelot	UNIX for Dummies Questions & Answers	11	01-30-2007 12:44 PM
Duplicate words	zulander	UNIX for Dummies Questions & Answers	1	04-01-2001 04:11 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Switch to Hybrid Mode Threaded Mode	Rate This Thread:


	Powered by