Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Remove repeated line using Perl

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove repeated line using Perl
# 1  
Old 09-10-2010
Remove repeated line using Perl
I am new to Perl and in text file of around 1000 lines having around 500 repeated line which I felt is no use and want to remove these line.so can somebody help in same for providing sample code how can i remove these repeated line in a file.


# 2  
Old 09-10-2010
Code:
sort -u

# 3  
Old 09-10-2010
here is one way to do it Remove Duplicate Lines from a File - Perl

do you have to use Perl? If not use the method that rdcwayx suggests.
Last edited by frank_rizzo; 09-10-2010 at 02:01 AM.. Reason: edit
# 4  
Old 09-10-2010
yes i have to use perl script.
so can somebody mail me sample code of same.
# 5  
Old 09-10-2010
Try
Code:
perl -i -ne 'next if $s{$_}++; print;' file

# 6  
Old 09-11-2010
After googling I found script for removing repeated line in a file.
Code:
next if $sLine =~ m/^\s*$/;  
$sLine=~s/^\s+//;
 $sLine=~s/\s+$//;  
print OUT qq{$sLine\n} unless ($hTmp{$sLine}++);

Basically regular expression are used to remove repeated line now Point is am I am not getting clear idea about these how they are working.Can somebody provide me some sort of help in that.
Last edited by pludi; 09-11-2010 at 07:03 PM..
# 7  
Old 09-11-2010
Quote:
Originally Posted by dinesh.4126
After googling I found script for removing repeated line in a file.
Code:
next if $sLine =~ m/^\s*$/;  
$sLine=~s/^\s+//;
 $sLine=~s/\s+$//;  
print OUT qq{$sLine\n} unless ($hTmp{$sLine}++);

Basically regular expression are used to remove repeated line now Point is am I am not getting clear idea about these how they are working....
The first three lines disregard (and do **not** print) blank lines and trim non-blank lines.
The fourth line is just another way of writing Klashxx's one-liner.
Try both of them with sample data and it should make things clearer.

tyler_durden
Last edited by pludi; 09-11-2010 at 07:03 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate lines which has been repeated 4 times

Remove duplicate lines which has been repeated 4 times attached test.txt below command tried and not getting expect output. for i in `cat test.txt | uniq` do num=`cat test.txt | grep $i | wc -l` echo $i $num done test.txt ... (17 Replies)
Discussion started by: Kalia
17 Replies

2. Shell Programming and Scripting

Remove repeated letter words

Hi, I have this text file with these words and I need help with removing words with repeated letter from these lines. 1 ama 5 bib 29 bob 2 bub 5 civic 2 dad 10 deed 1 denned 335 did 1 eeee 1 eeeee 2 eke 8... (4 Replies)
Discussion started by: crepe6
4 Replies

3. Shell Programming and Scripting

Find repeated word and take sum of the second field to it ,for all the repeated words in awk

Hi below is the input file, i need to find repeated words and sum up the values of it which is second field from the repeated work.Im trying but getting no where close to it.Kindly give me a hint on how to go about it Input fruits,apple,20,fruits,mango,20,veg,carrot,12,veg,raddish,30... (11 Replies)
Discussion started by: 100bees
11 Replies

4. Shell Programming and Scripting

Need to remove first 6 lines and last line in a array ---- perl scripting

Hi I have stored a command output in an array like below @a = `xyz`; actually xyz comnad will give the output like this tracker date xxxxxxx xxxxxxx --------------------- 1 a 2 b ---------------------- i have stored the "xyz" output to an... (3 Replies)
Discussion started by: siva kumar
3 Replies

5. Shell Programming and Scripting

Remove regularly repeated lines

How can i delete some regular repeated lines in a file? example: in_file EDGE 1 2 12 EDGE 2 3 23 EDGE 3 4 34 EDGE 5 6 56 EDGE 6 7 67 EDGE 7 8 78 EDGE 9 10 910 EDGE 10 11 1011 EDGE 11 12 1112 EDGE 13 14 1314 EDGE 14 15 1415 EDGE 15 16 1516 EDGE 17 18 1718 EDGE 18 19 1819 EDGE 19... (8 Replies)
Discussion started by: saeed.soltani
8 Replies

6. Shell Programming and Scripting

ksh Remove and replace repeated in file

Hi, i need to read a line from a file and count the number of times it appear in, then continuous to the second line with the same. So when i count a line i have to remove all duplicates in the file to not count it another time. while read line do n=$(grep -c $line File) print "$line... (5 Replies)
Discussion started by: ToniX
5 Replies

7. Shell Programming and Scripting

Perl regex to remove a segment in a line

Hello, ksh on Sun5.8 here. I have a pipe-delimited, variable length record file with sub-segments identified with a tilda that we receive from a source outside of our control. The records are huge, and Perl seems to be the only shell that can handle the huge lines. I am new to Perl, and am... (8 Replies)
Discussion started by: gary_w
8 Replies

8. Shell Programming and Scripting

remove brackets and put it in a column and remove repeated entry

Hi all, I want to remove the remove bracket sign ( ) and put in the separate column I also want to remove the repeated entry like in first row in below input (PA156) is repeated ESR1 (PA156) leflunomide (PA450192) (PA156) leflunomide (PA450192) CHST3 (PA26503) docetaxel... (2 Replies)
Discussion started by: manigrover
2 Replies

9. Shell Programming and Scripting

remove anything after repeated string pattern found

HI, Can anyone help me with a script. i/p calc 1 2 3 4 5 6 7 8 calc 4 5 6 calc 7 8 9 o/p calc 1 2 3 4 5 6 7 8 calc 4 5 6 i.e remove anything after where two times the string calc is found. thanks (3 Replies)
Discussion started by: Indra2011
3 Replies

10. Shell Programming and Scripting

how to remove line from /etc/vfstab using shell / perl

Hi, could someone help me on this i want to remove line from /etc/vfstab in the system how to do that it is rite now like this /dev/vx/dsk/appdg1/mytestvol /dev/vx/rdsk/appdg1/mytestvol /mytest vxfs 3 no largefiles /dev/vx/dsk/appdg1/mytestvol1 ... (2 Replies)
Discussion started by: tarunn.dubeyy
2 Replies
Login or Register to Ask a Question