Sponsored Content
Top Forums Shell Programming and Scripting Remove duplicate lines from file based on fields Post 302926698 by Lord Spectre on Wednesday 26th of November 2014 08:01:26 AM
Old 11-26-2014
Remove duplicate lines from file based on fields

Dear community,
I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns

The data are like this:
Code:
Region           23/11/2014 09:11:36 41752
Medio           23/11/2014 03:11:38 4132
Info                 23/11/2014 05:11:09 4323
Test                 23/11/2014 05:11:14 4323
Info                 23/11/2014 07:11:09 4323
Test2                 23/11/2014 08:11:14 4323

In that case I need to remove one of the line who contains "Info" and "4323". So the output will be:
Code:
Region           23/11/2014 09:11:36 41752
Medio           23/11/2014 03:11:38 4132
Info                 23/11/2014 05:11:09 4323
Test                 23/11/2014 05:11:14 4323
Test2                 23/11/2014 08:11:14 4325

Thanks
Lucas
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. UNIX for Dummies Questions & Answers

Remove duplicate rows of a file based on a value of a column

Hi, I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g. COL1 COL2 COL3 A 1234 1234 B 3k32 2322 C Xk32 TTT A NEW XX22 B 3k32 ... (7 Replies)
Discussion started by: risk_sly
7 Replies

4. UNIX for Dummies Questions & Answers

How to get remove duplicate of a file based on many conditions

Hii Friends.. I have a huge set of data stored in a file.Which is as shown below a.dat: RAO 1869 12 19 0 0 0.00 17.9000 82.3000 10.0 0 0.00 0 3.70 0.00 0.00 0 0.00 3.70 4 NULL LEE 1870 4 11 1 0 0.00 30.0000 99.0000 0.0 0 0.00 0 0.00 0.00 0.00 0 ... (3 Replies)
Discussion started by: reva
3 Replies

5. UNIX for Dummies Questions & Answers

remove duplicate lines based on two columns and judging from a third one

hello all, I have an input file with four columns like this with a lot of lines and for example, line 1 and line 5 match because the first 4 characters match and the fourth column matches too. I want to keep the line that has the lowest number in the third column. So I discard line 5.... (5 Replies)
Discussion started by: TheTransporter
5 Replies

6. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

7. Shell Programming and Scripting

Remove duplicate lines from a 50 MB file size

hi, Please help me to write a command to delete duplicate lines from a file. And the size of file is 50 MB. How to remove duplicate lins from such a big file. (6 Replies)
Discussion started by: vsachan
6 Replies

8. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

9. Shell Programming and Scripting

Remove duplicate lines based on field and sort

I have a csv file that I would like to remove duplicate lines based on field 1 and sort. I don't care about any of the other fields but I still wanna keep there data intact. I was thinking I could do something like this but I have no idea how to print the full line with this. Please show any method... (8 Replies)
Discussion started by: cokedude
8 Replies

10. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies
Test::Moose(3)						User Contributed Perl Documentation					    Test::Moose(3)

NAME
Test::Moose - Test functions for Moose specific features VERSION
version 2.1202 SYNOPSIS
use Test::More plan => 1; use Test::Moose; meta_ok($class_or_obj, "... Foo has a ->meta"); does_ok($class_or_obj, $role, "... Foo does the Baz role"); has_attribute_ok($class_or_obj, $attr_name, "... Foo has the 'bar' attribute"); DESCRIPTION
This module provides some useful test functions for Moose based classes. It is an experimental first release, so comments and suggestions are very welcome. EXPORTED FUNCTIONS
meta_ok ($class_or_object) Tests if a class or object has a metaclass. does_ok ($class_or_object, $role, ?$message) Tests if a class or object does a certain role, similar to what "isa_ok" does for the "isa" method. has_attribute_ok($class_or_object, $attr_name, ?$message) Tests if a class or object has a certain attribute, similar to what "can_ok" does for the methods. with_immutable { CODE } @class_names Runs CODE (which should contain normal tests) twice, and make each class in @class_names immutable in between the two runs. The CODE block is called with a single boolean argument indicating whether or not the classes have been made immutable yet. TODO
Convert the Moose test suite to use this module. Here is a list of possible functions to write immutability predicates anon-class predicates discovering original method from modified method attribute metaclass predicates (attribute_isa?) SEE ALSO
Test::More BUGS
See "BUGS" in Moose for details on reporting bugs. AUTHORS
o Stevan Little <stevan.little@iinteractive.com> o Dave Rolsky <autarch@urth.org> o Jesse Luehrs <doy@tozt.net> o Shawn M Moore <code@sartak.org> o XXXX XXX'XX (Yuval Kogman) <nothingmuch@woobling.org> o Karen Etheridge <ether@cpan.org> o Florian Ragwitz <rafl@debian.org> o Hans Dieter Pearcey <hdp@weftsoar.net> o Chris Prather <chris@prather.org> o Matt S Trout <mst@shadowcat.co.uk> COPYRIGHT AND LICENSE
This software is copyright (c) 2006 by Infinity Interactive, Inc.. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. perl v5.18.2 2014-01-19 Test::Moose(3)
All times are GMT -4. The time now is 04:08 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy