Perl, sorting and eliminating duplicates


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Perl, sorting and eliminating duplicates
# 1  
Old 06-04-2012
Perl, sorting and eliminating duplicates

Hi guys! I'm trying to eliminate some duplicates from a file but I'm like this Smilie !!!

My file looks like this:
Code:
ID_1  0.02
ID_2  2.4e-2
ID_2  4.3.e-9
ID_3  0.003
ID_4  0.2
ID_4  0.05
ID_5  1.2e-3

What I need is to eliminate all the duplicates considering the first column (in this example an ID_2 and an ID_4) but I would like to eliminate all the duplicates but one considering the value in column 2. Particularly where there are some duplicates I want to maintain the one with the lower value associated. In this case I would like to eliminate ID_2 2.4e-2 and ID_4 0.2.

Someone can help me in some ways?!!?

Thank you!!!
cheers

Moderator's Comments:
Mod Comment Please use next time code tags for your code and data

Last edited by vbe; 06-04-2012 at 11:37 AM..
# 2  
Old 06-04-2012
Something like the following should work, WARNING not tested code.

Code:
#!/usr/bin/perl

use strict;
use warnings;
my %values;
open (my $file, "<", $ARGV[0]);
while(<file>){
  my ($id,$val)=/(ID_\d+)\s+(.+)$/;
  if (! defined $values{$id}){
    $values{$id}=sprintf ("%.15f\n",$val);
  }
  elsif (sprintf ("%.15f\n",$val) < $values{$id}){
    $values{$id}=sprintf ("%.15f\n",$val);
  }
}
for (sort keys %values){
  print "$_\t$values{$_};
}

# 3  
Old 06-04-2012
I tryed it, but I get an error like this:
Code:
Unknown open () mode 'file.txt' at perl.pl line 4

.

?

Last edited by Scrutinizer; 06-04-2012 at 01:03 PM.. Reason: code tags
# 4  
Old 06-04-2012
Code:
[root@host dir]# cat input
ID_1  0.02
ID_2  2.4e-2
ID_2  4.3e-9
ID_3  0.003
ID_4  0.2
ID_4  0.05
ID_5  1.2e-3
[root@host dir]#
[root@host dir]# perl -lane '
if (defined $x{$F[0]}) {
    if ($x{$F[0]} > $F[1]) {
        $x{$F[0]} = $F[1];
    }
}
else {
    $x{$F[0]} = $F[1];
}
END {
    for (sort keys %x) { print "$_ $x{$_}" }
}' input
ID_1 0.02
ID_2 4.3e-9
ID_3 0.003
ID_4 0.05
ID_5 1.2e-3
[root@host dir]#

# 5  
Old 06-04-2012
A simpler way in awk:

Code:
$ awk '$1 in A { if(($2+0) > A[$1]) next } { A[$1]=$2+0; B[$1]=$2 } END { for(X in B) print X, B[X] }' data
ID_1 0.02
ID_2 2.4e-2
ID_3 0.003
ID_4 0.05
ID_5 1.2e-3

$


Last edited by Corona688; 06-04-2012 at 02:32 PM..
# 6  
Old 06-04-2012
With GNU sort
Code:
sort -k1,1 -k2,2g input-file |sort -msu -k1,1

# 7  
Old 06-05-2012
thank you all! very nice suggestions!!
the awk works perfectly for my case! thank you!! :-)
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help in modifying a PERL script to sort Singletons and Duplicates

I have a large database which has the following structure a=b where a is one language and b is the other and = is the delimiter Since the data treats of language, homographs occur i.e. the same word on the left hand side can map in two different entries to two different glosses on the right... (3 Replies)
Discussion started by: gimley
3 Replies

2. Shell Programming and Scripting

PERL "filtering the log file removing the duplicates

Hi folks, I have a log file in the below format and trying to get the output of the unique ones based on mnemonic IN PERL. Could any one please let me know with the code and the logic ? Severity Mnemonic Log Message 7 CLI_SCHEDULER Logfile for scheduled CLI... (3 Replies)
Discussion started by: scriptscript
3 Replies

3. Shell Programming and Scripting

Perl Sorting Help

Hey guys, I have started to learn perl recently because of a position I took. They want me to master perl and I've been reading books and practicing myself. Basically I,m having my perl script run through a text pad and give the output in a special way e.g input deviceconfig { ... (5 Replies)
Discussion started by: zee3b
5 Replies

4. Shell Programming and Scripting

Need help with eliminating newlines with Perl

Good morning, I need some help with getting rid of newlines with the output from a MYSQL query and putting the information into the right format that I need. Here is the script as it is today: #!/usr/bin/perl my $uda = system("/opt/incontrol/mysql/bin/mysql -u root -ppassword... (2 Replies)
Discussion started by: brianjb
2 Replies

5. Shell Programming and Scripting

HELP on Perl array / sorting - trying to convert Korn Shell Script to Perl

Hi all, Not sure if this should be in the programming forum, but I believe it will get more response under the Shell Programming and Scripting FORUM. Am trying to write a customized df script in Perl and need some help with regards to using arrays and file handlers. At the moment am... (3 Replies)
Discussion started by: newbie_01
3 Replies

6. Shell Programming and Scripting

Perl sorting

Hi, I have a file in this format: a b c d e a b c d e a b c d e i need perl script to sort 2nd column in alphabatical order The script i use is #!/usr/bin/perl my @words=<>; foreach(sort mysort @words) { print; (4 Replies)
Discussion started by: Learnerabc
4 Replies

7. Shell Programming and Scripting

Eliminating duplicates from the who command

I am trying to show how many users are logged into one of my systems. Using who -q it gives me a user count, but some users are logged in multiple times. Is there any easy parameter that I can use to ignore/eliminate these duplicates?? Thanks (9 Replies)
Discussion started by: mikejreading
9 Replies

8. Shell Programming and Scripting

perl sorting

I have many files that I need to sort each week. I know how to do in Unix, but for this task it appears best to do native inside an existing perl program. So, simplified, I have a file similar to the following: Joe_________12_Main_St__A001________LX Benny_______5_Spring____A002________LX... (5 Replies)
Discussion started by: joeyg
5 Replies

9. UNIX for Dummies Questions & Answers

How to remove duplicates without sorting

Hello, I can remove duplicate entries in a file by: sort File1 | uniq > File2 but how can I remove duplicates without sorting the file? I tried cat File1 | uniq > File2 but it doesn't work thanks (4 Replies)
Discussion started by: orahi001
4 Replies

10. Shell Programming and Scripting

finding duplicates with perl

I have a huge file (over 30mb) that I am processing through with perl. I am pulling out a list of filenames and placing it in an array called @reports. I am fine up till here. What I then want to do is go through the array and find any duplicates. If there is a duplicate, output it to the screen.... (3 Replies)
Discussion started by: dangral
3 Replies
Login or Register to Ask a Question