Sponsored Content
Top Forums Shell Programming and Scripting How to delete duplicate records based on key Post 302268400 by radoulov on Monday 15th of December 2008 03:26:27 PM
Old 12-15-2008
Perl:

Code:
perl -F, -lane'
  $u{$F[0]}++; $r{$F[0]} = $_;
  print join $/, map $r{$_}, 
    grep $u{$_} == 1, 
      sort {$a<=>$b} keys %u if eof
  ' infile

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete Duplicate records from a tilde delimited file

Hi All, I want to delete duplicate records from a tilde delimited file. Criteria is considering the first 2 fields, the combination of which has to be unique, below is a sample of records in the input file 1620000010338~2446694087~0~20061130220000~A00BCC1CT... (5 Replies)
Discussion started by: irshadm
5 Replies

2. Shell Programming and Scripting

how to delete duplicate rows based on last column

hii i have a huge amt of data stored in a file.Here in this file i need to remove duplicates rows in such a way that the last column has different data & i must check for greatest among last colmn data & print the largest data along with other entries but just one of other duplicate entries is... (16 Replies)
Discussion started by: reva
16 Replies

3. UNIX for Dummies Questions & Answers

Delete lines with duplicate strings based on date

Hey all, a relative bash/script newbie trying solve a problem. I've got a text file with lots of lines that I've been able to clean up and format with awk/sed/cut, but now I'd like to remove the lines with duplicate usernames based on time stamp. Here's what the data looks like 2007-11-03... (3 Replies)
Discussion started by: mattv
3 Replies

4. UNIX for Dummies Questions & Answers

forming duplicate rows based on value of a key

if the key (A or B or ...others) has 4 in its 3rd column the 1st A row has to form 4 dupicates along with the all the values of A in 4th column (2.9, 3.8, 4.2) . Hope I explain the question clearly. Cheers Ruby input "A" 1 4 2.9 "A" 2 5 ... (7 Replies)
Discussion started by: ruby_sgp
7 Replies

5. Shell Programming and Scripting

awk - splitting 1 large file into multiple based on same key records

Hello gurus, I am new to "awk" and trying to break a large file having 4 million records into several output files each having half million but at the same time I want to keep the similar key records in the same output file, not to exist accross the files. e.g. my data is like: Row_Num,... (6 Replies)
Discussion started by: kam66
6 Replies

6. Shell Programming and Scripting

Removing duplicate records in a file based on single column

Hi, I want to remove duplicate records including the first line based on column1. For example inputfile(filer.txt): ------------- 1,3000,5000 1,4000,6000 2,4000,600 2,5000,700 3,60000,4000 4,7000,7777 5,999,8888 expected output: ---------------- 3,60000,4000 4,7000,7777... (5 Replies)
Discussion started by: G.K.K
5 Replies

7. Shell Programming and Scripting

Removing duplicate records in a file based on single column explanation

I was reading this thread. It looks like a simpler way to say this is to only keep uniq lines based on field or column 1. https://www.unix.com/shell-programming-scripting/165717-removing-duplicate-records-file-based-single-column.html Can someone explain this command please? How are there no... (5 Replies)
Discussion started by: cokedude
5 Replies

8. UNIX for Dummies Questions & Answers

Delete records from a big file based on some condition

Hi, To load a big file in a table,I have a make sure that all rows in the file has same number of the columns . So in my file if I am getting any rows which have columns not equal to 6 , I need to delete it . Delimiter is space and columns are optionally enclosed by "". This can be ... (1 Reply)
Discussion started by: hemantraijain
1 Replies

9. Shell Programming and Scripting

Removing specific records from files when duplicate key

Hello I have been trying to remove a row from a file which has the same first three columns as another row - I have tried lots of different combinations of suggestion on this forum but can't get it exactly right. what I have is 900 - 1000 = 0 900 - 1000 = 2562 1000 - 1100 = 0 1000 - 1100... (7 Replies)
Discussion started by: tinytimmay
7 Replies

10. Shell Programming and Scripting

Delete duplicate row based on criteria

Hi, I have an input file as shown below: 20140102;13:30;FR-AUD-LIBOR-1W;2.495 20140103;13:30;FR-AUD-LIBOR-1W;2.475 20140106;13:30;FR-AUD-LIBOR-1W;2.495 20140107;13:30;FR-AUD-LIBOR-1W;2.475 20140108;13:30;FR-AUD-LIBOR-1W;2.475 20140109;13:30;FR-AUD-LIBOR-1W;2.475... (2 Replies)
Discussion started by: shash
2 Replies
Mojo::Collection(3pm)					User Contributed Perl Documentation				     Mojo::Collection(3pm)

NAME
Mojo::Collection - Collection SYNOPSIS
# Manipulate collections use Mojo::Collection; my $collection = Mojo::Collection->new(qw(just works)); unshift @$collection, 'it'; $collection->map(sub { ucfirst })->each(sub { my ($word, $count) = @_; say "$count: $word"; }); # Use the alternative constructor use Mojo::Collection 'c'; c(qw(a b c))->join('/')->url_escape->say; DESCRIPTION
Mojo::Collection is a container for collections. FUNCTIONS
Mojo::Collection implements the following functions. "c" my $collection = c(1, 2, 3); Construct a new Mojo::Collection object. METHODS
Mojo::Collection implements the following methods. "new" my $collection = Mojo::Collection->new(1, 2, 3); Construct a new Mojo::Collection object. "each" my @elements = $collection->each; $collection = $collection->each(sub {...}); Evaluate closure for each element in collection. $collection->each(sub { my ($e, $count) = @_; say "$count: $e"; }); "first" my $first = $collection->first; my $first = $collection->first(sub {...}); Evaluate closure for each element in collection and return the first one for which the closure returns true. my $five = $collection->first(sub { $_ == 5 }); "grep" my $new = $collection->grep(sub {...}); Evaluate closure for each element in collection and create a new collection with all elements for which the closure returned true. my $interesting = $collection->grep(sub { /mojo/i }); "join" my $stream = $collection->join(" "); Turn collection into Mojo::ByteStream. $collection->join(" ")->say; "map" my $new = $collection->map(sub {...}); Evaluate closure for each element in collection and create a new collection from the results. my $doubled = $collection->map(sub { $_ * 2 }); "reverse" my $new = $collection->reverse; Create a new collection with all elements in reverse order. "slice" my $new = $collection->slice(4 .. 7); Create a new collection with all selected elements. "shuffle" my $new = $collection->shuffle; Create a new collection with all elements in random order. "size" my $size = $collection->size; Number of elements in collection. "sort" my $new = $collection->sort; my $new = $collection->sort(sub {...}); Sort elements based on return value of closure and create a new collection from the results. my $insensitive = $collection->sort(sub { uc(shift) cmp uc(shift) }); SEE ALSO
Mojolicious, Mojolicious::Guides, <http://mojolicio.us>. perl v5.14.2 2012-09-05 Mojo::Collection(3pm)
All times are GMT -4. The time now is 03:21 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy