Sponsored Content
Top Forums Shell Programming and Scripting Remove duplicate lines from file based on fields Post 302926698 by Lord Spectre on Wednesday 26th of November 2014 08:01:26 AM
Old 11-26-2014
Remove duplicate lines from file based on fields

Dear community,
I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns

The data are like this:
Code:
Region           23/11/2014 09:11:36 41752
Medio           23/11/2014 03:11:38 4132
Info                 23/11/2014 05:11:09 4323
Test                 23/11/2014 05:11:14 4323
Info                 23/11/2014 07:11:09 4323
Test2                 23/11/2014 08:11:14 4323

In that case I need to remove one of the line who contains "Info" and "4323". So the output will be:
Code:
Region           23/11/2014 09:11:36 41752
Medio           23/11/2014 03:11:38 4132
Info                 23/11/2014 05:11:09 4323
Test                 23/11/2014 05:11:14 4323
Test2                 23/11/2014 08:11:14 4325

Thanks
Lucas
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. UNIX for Dummies Questions & Answers

Remove duplicate rows of a file based on a value of a column

Hi, I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g. COL1 COL2 COL3 A 1234 1234 B 3k32 2322 C Xk32 TTT A NEW XX22 B 3k32 ... (7 Replies)
Discussion started by: risk_sly
7 Replies

4. UNIX for Dummies Questions & Answers

How to get remove duplicate of a file based on many conditions

Hii Friends.. I have a huge set of data stored in a file.Which is as shown below a.dat: RAO 1869 12 19 0 0 0.00 17.9000 82.3000 10.0 0 0.00 0 3.70 0.00 0.00 0 0.00 3.70 4 NULL LEE 1870 4 11 1 0 0.00 30.0000 99.0000 0.0 0 0.00 0 0.00 0.00 0.00 0 ... (3 Replies)
Discussion started by: reva
3 Replies

5. UNIX for Dummies Questions & Answers

remove duplicate lines based on two columns and judging from a third one

hello all, I have an input file with four columns like this with a lot of lines and for example, line 1 and line 5 match because the first 4 characters match and the fourth column matches too. I want to keep the line that has the lowest number in the third column. So I discard line 5.... (5 Replies)
Discussion started by: TheTransporter
5 Replies

6. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

7. Shell Programming and Scripting

Remove duplicate lines from a 50 MB file size

hi, Please help me to write a command to delete duplicate lines from a file. And the size of file is 50 MB. How to remove duplicate lins from such a big file. (6 Replies)
Discussion started by: vsachan
6 Replies

8. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

9. Shell Programming and Scripting

Remove duplicate lines based on field and sort

I have a csv file that I would like to remove duplicate lines based on field 1 and sort. I don't care about any of the other fields but I still wanna keep there data intact. I was thinking I could do something like this but I have no idea how to print the full line with this. Please show any method... (8 Replies)
Discussion started by: cokedude
8 Replies

10. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies
Locale::Language(3pm)					 Perl Programmers Reference Guide				     Locale::Language(3pm)

NAME
Locale::Language - standard codes for language identification SYNOPSIS
use Locale::Language; $lang = code2language('en'); # $lang gets 'English' $code = language2code('French'); # $code gets 'fr' @codes = all_language_codes(); @names = all_language_names(); DESCRIPTION
The "Locale::Language" module provides access to standard codes used for identifying languages, such as those as defined in ISO 639. Most of the routines take an optional additional argument which specifies the code set to use. If not specified, the default ISO 639 two- letter codes will be used. SUPPORTED CODE SETS
There are several different code sets you can use for identifying languages. A code set may be specified using either a name, or a constant that is automatically exported by this module. For example, the two are equivalent: $lang = code2language('en','alpha-2'); $lang = code2language('en',LOCALE_CODE_ALPHA_2); The codesets currently supported are: alpha-2, LOCALE_LANG_ALPHA_2 This is the set of two-letter (lowercase) codes from ISO 639-1, such as 'he' for Hebrew. It also includes additions to this set included in the IANA language registry. This is the default code set. alpha-3, LOCALE_LANG_ALPHA_3 This is the set of three-letter (lowercase) bibliographic codes from ISO 639-2 and 639-5, such as 'heb' for Hebrew. It also includes additions to this set included in the IANA language registry. term, LOCALE_LANG_TERM This is the set of three-letter (lowercase) terminologic codes from ISO 639. ROUTINES
code2language ( CODE [,CODESET] ) language2code ( NAME [,CODESET] ) language_code2code ( CODE ,CODESET ,CODESET2 ) all_language_codes ( [CODESET] ) all_language_names ( [CODESET] ) Locale::Language::rename_language ( CODE ,NEW_NAME [,CODESET] ) Locale::Language::add_language ( CODE ,NAME [,CODESET] ) Locale::Language::delete_language ( CODE [,CODESET] ) Locale::Language::add_language_alias ( NAME ,NEW_NAME ) Locale::Language::delete_language_alias ( NAME ) Locale::Language::rename_language_code ( CODE ,NEW_CODE [,CODESET] ) Locale::Language::add_language_code_alias ( CODE ,NEW_CODE [,CODESET] ) Locale::Language::delete_language_code_alias ( CODE [,CODESET] ) These routines are all documented in the Locale::Codes::API man page. SEE ALSO
Locale::Codes The Locale-Codes distribution. Locale::Codes::API The list of functions supported by this module. http://www.loc.gov/standards/iso639-2/ Source of the ISO 639-2 codes. http://www.loc.gov/standards/iso639-5/ Source of the ISO 639-5 codes. http://www.iana.org/assignments/language-subtag-registry The IANA language subtag registry. AUTHOR
See Locale::Codes for full author history. Currently maintained by Sullivan Beck (sbeck@cpan.org). COPYRIGHT
Copyright (c) 1997-2001 Canon Research Centre Europe (CRE). Copyright (c) 2001-2010 Neil Bowers Copyright (c) 2010-2013 Sullivan Beck This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.18.2 2014-01-06 Locale::Language(3pm)
All times are GMT -4. The time now is 03:43 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy