Eliminating duplicate lines via specified number of digits Post: 302824531

Login or Register to Ask a Question and Join Our Community

Sponsored Content

Top Forums Shell Programming and Scripting Eliminating duplicate lines via specified number of digits Post 302824531 by Fundix on Friday 21st of June 2013 05:02:22 AM

Old

06-21-2013

Registered User

A Perl solution using 2 hastables :

Code:

#!/usr/bin/perl -w
use strict;

my $cur_dir = $ENV{PWD};
my $filename = "$cur_dir/$ARGV[0]";
my ($record,@fields,$prefix,%peers,$key,%records);

open(FILEIN,"<$filename") or die"open: $!";
while( defined( $record = <FILEIN> ) ) {
  chomp $record;

  @fields=split(/ /,$record);
  $prefix=substr($fields[0],0,3);

  if(! exists( $peers{$prefix} ) || $fields[1] < $peers{$prefix} ) {
    $peers{$prefix} = $fields[1];
    $records{$prefix} = $record;
  }
}
close(FILEIN);

foreach my $key (sort keys(%records)) {
   print "$records{$key}\n";
}

output :

Code:

%./file031.pl file031
123 45.34
345-xx 32.56
888 22.33
949-xx 36.55

Last edited by Fundix; 06-21-2013 at 06:03 AM.. Reason: Missing }

Fundix

View Public Profile for Fundix

Find all posts by Fundix

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

restrain the number of digits of a PID

How is it possible under UNIX to restrain the number of digits of the PID number? For instance, we have a product that generates a PID of 7 digits, and we would like to have only 6 digits maximum instead for the PID. Thank you for your help.

2. UNIX for Dummies Questions & Answers

Counting The Number Of Duplicate Lines In a File

Hello. First time poster here. I have a huge file of IP numbers. I am trying to output only the class b of the IPs and rank them by most common and output the total # of duplicate class b's before the class b. An example is below: 12.107.1.1 12.107.9.54 12.108.3.89 12.109.109.4 12.109.6.3 ...

3. Shell Programming and Scripting

Count number of digits in a word

Hi all Can anybody suggest me, how to get the count of digits in a word I tried WORD=abcd1234 echo $WORD | grep -oE ] | wc -l 4 It works in bash command line, but not in scripts :mad:

4. UNIX for Dummies Questions & Answers

Eliminating CR (new lines) from a file.

Hi all, I made a C++ program in dos (in dev-C++) and uploaded it on Solaris box. On opening that file with 'vim' editor i found that there is some extra new lines after each written code line. I tried to find out is the file is in dos or in unix format, with 'file' command,and i got "<file-name>.h:...

5. Shell Programming and Scripting

number of digits after decimal

Hi All, I have a file of decimal numbers, cat file1.txt 1.1382666907 1.2603107334 1.6118799297 24.4995857056 494.7632588468 560.7633734425 ..... I want to see the output as only 7 digits after decimal

6. Shell Programming and Scripting

summing the digits of a binary nuMBER

please help me write a perl program to find the difference of 1 and zeros of a 6 digit binary number. eg If input is 111100 expected output +2 if input is 000011 expected output -2 input is 000111 expected output 0

7. Shell Programming and Scripting

extracting Number variable and the following digits.

HI all, I have output of something like this: crab: ExitCodes Summary >>>>>>>>> 12 Jobs with Wrapper Exit Code : 50117 List of jobs: 1-12 See https:///twiki/something/ for Exit Code meaning crab: ExitCodes Summary >>>>>>>>> 5 Jobs with Wrapper Exit Code : 8001 List of...

8. Shell Programming and Scripting

Eliminating duplicate lines

Hello, I am trying to eliminate lines where column #1 is duplicated. If it is a duplicate, the line with the greater value in column #2 should be deleted: file.dat 123 45.34 345 67.22 949 36.55 123 94.23 888 22.33 345 32.56 Desired ouput 123 45.34 949 36.55 888 22.33 345 32.56...

9. Shell Programming and Scripting

awk changes to cut number of digits

HCPM1ONDB00014800011800000589009211201 L201307022013070228AUD 00000000031. 000965105800000000000000000000000 MOBITV KEYA ...

10. Shell Programming and Scripting

Find number of digits in a word

HI, Can you tell me how to find the number of digits in a word. $cat data.txt +123456ad 87645768 Output should be 6 8

LEARN ABOUT DEBIAN

findbin

FindBin(3perl)						 Perl Programmers Reference Guide					    FindBin(3perl)

NAME

       FindBin - Locate directory of original perl script

SYNOPSIS

	use FindBin;
	use lib "$FindBin::Bin/../lib";

	or

	use FindBin qw($Bin);
	use lib "$Bin/../lib";

DESCRIPTION

       Locates the full path to the script bin directory to allow the use of paths relative to the bin directory.

       This allows a user to setup a directory tree for some software with directories "<root>/bin" and "<root>/lib", and then the above example
       will allow the use of modules in the lib directory without knowing where the software tree is installed.

       If perl is invoked using the -e option or the perl script is read from "STDIN" then FindBin sets both $Bin and $RealBin to the current
       directory.

EXPORTABLE VARIABLES

	$Bin	     - path to bin directory from where script was invoked
	$Script      - basename of script from which perl was invoked
	$RealBin     - $Bin with all links resolved
	$RealScript  - $Script with all links resolved

KNOWN ISSUES

       If there are two modules using "FindBin" from different directories under the same interpreter, this won't work. Since "FindBin" uses a
       "BEGIN" block, it'll be executed only once, and only the first caller will get it right. This is a problem under mod_perl and other
       persistent Perl environments, where you shouldn't use this module. Which also means that you should avoid using "FindBin" in modules that
       you plan to put on CPAN. To make sure that "FindBin" will work is to call the "again" function:

	 use FindBin;
	 FindBin::again(); # or FindBin->again;

       In former versions of FindBin there was no "again" function. The workaround was to force the "BEGIN" block to be executed again:

	 delete $INC{'FindBin.pm'};
	 require FindBin;

KNOWN BUGS

       If perl is invoked as

	  perl filename

       and filename does not have executable rights and a program called filename exists in the users $ENV{PATH} which satisfies both -x and -T
       then FindBin assumes that it was invoked via the $ENV{PATH}.

       Workaround is to invoke perl as

	perl ./filename

AUTHORS

       FindBin is supported as part of the core perl distribution. Please send bug reports to <perlbug@perl.org> using the perlbug program
       included with perl.

       Graham Barr <gbarr@pobox.com> Nick Ing-Simmons <nik@tiuk.ti.com>

COPYRIGHT

       Copyright (c) 1995 Graham Barr & Nick Ing-Simmons. All rights reserved.	This program is free software; you can redistribute it and/or
       modify it under the same terms as Perl itself.

perl v5.14.2							    2011-09-26							    FindBin(3perl)