Sponsored Content
Top Forums UNIX for Advanced & Expert Users Performance problem with removing duplicates in a huge file (50+ GB) Post 302752651 by Kannan K on Monday 7th of January 2013 10:55:16 AM
Old 01-07-2013
Sample Records

Sample records from file:

Code:
14480020180,A20180,A020180,143245765381,A00062,17284171796 
14480020180,A20180,A020180,143245765381,A00062,17284171796 
14480000127,A00127,A000127,143245730649,A00127, 
14480020180,A20180,A020180,143245765381,A00062,17284171796 
14480000127,A00127,A000127,143245730649,A00127, 
14480020180,A20180,A020180,143245765381,A00062,17284171796 
14480042302,A42302,A000127,143245800913,A00127, 
14480020180,A20180,A020180,143245765381,A00062,17284171796 
14480041999,A41999,A000127,143245801337,A00127, 
14480020180,A20180,A020180,143245765381,A00062,17284171796 
14480000163,A00163,A000163,143245730774,A00163,4133403 
14480042302,A42302,A000127,143245800913,A00127,

Desired Output:-
Code:
14480020180,A20180,A020180,143245765381,A00062,17284171796 
14480000127,A00127,A000127,143245730649,A00127, 
14480000163,A00163,A000163,143245730774,A00163,4133403 
14480041999,A41999,A000127,143245801337,A00127, 
14480042302,A42302,A000127,143245800913,A00127,

I also want to add the fact that this file contains 40-50% (20-25 GB) of duplicate records.
And unfortunately, all columns need to considered as part of the key to determine duplicates.

The order of the data (sorted/unsorted) in the resultant file doesn't matter. Only the removal of duplicates is essential.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

removing duplicates from a file

i have a file with some 1000 entries it will contain entries like 1000,ram 2000,pankaj 1001,rahim 1000,ram 2532,govind 2000,pankaj 3000,venkat 2532,govind what i want is i want to extract only the distinct rows from this file so my output should contain only 1000,ram... (2 Replies)
Discussion started by: trichyselva
2 Replies

2. UNIX for Dummies Questions & Answers

removing duplicates of a pattern from a file

hey all, I need some help. I have a text file with names in it. My target is that if a particular pattern exists in that file more than once..then i want to rename all the occurences of that pattern by alternate patterns.. for e.g if i have PATTERN occuring 5 times then i want to... (3 Replies)
Discussion started by: ashisharora
3 Replies

3. Shell Programming and Scripting

Removing duplicates from log file?

I have a log file with posts looking like this: -- Messages can be delivered by different systems at different times. The id number is used to sort out duplicate messages. What I need is to strip the arrival time from each post, sort posts by id number, and reattach arrival time to respective... (2 Replies)
Discussion started by: Ilja
2 Replies

4. Shell Programming and Scripting

Removing Duplicates from file

Hi Experts, Please check the following new requirement. I got data like the following in a file. FILE_HEADER 01cbbfde7898410| 3477945| home| 1 01cbc275d2c122| 3478234| WORK| 1 01cbbe4362743da| 3496386| Rich Spare| 1 01cbc275d2c122| 3478234| WORK| 1 This is pipe separated file with... (3 Replies)
Discussion started by: tinufarid
3 Replies

5. Shell Programming and Scripting

formatting a file and removing duplicates

Hi, I have a file that I want to change the format of. It is a large file in rows but I want it to be comma separated (comma then a space). The current file looks like this: HI, Joe, Bob, Jack, Jack After I would want to remove any duplicates so it would look like this: HI, Joe,... (2 Replies)
Discussion started by: kylle345
2 Replies

6. HP-UX

Performance issue with 'grep' command for huge file size

I have 2 files; one file (say, details.txt) contains the details of employees and another file (say, emp.txt) has some selected employee names. I am extracting employee details from details.txt by using emp.txt and the corresponding code is: while read line do emp_name=`echo $line` grep -e... (7 Replies)
Discussion started by: arb_1984
7 Replies

7. UNIX for Dummies Questions & Answers

Removing duplicates from a file

Hi All, I am merging files coming from 2 different systems ,while doing that I am getting duplicates entries in the merged file I,01,000131,764,2,4.00 I,01,000131,765,2,4.00 I,01,000131,772,2,4.00 I,01,000131,773,2,4.00 I,01,000168,762,2,2.00 I,01,000168,763,2,2.00... (5 Replies)
Discussion started by: Sri3001
5 Replies

8. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3 (2 Replies)
Discussion started by: sagar_1986
2 Replies

9. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3.I have tried previous post also,but in that complete line must be similar.In this case i have to verify first column only regardless what is the content in succeeding columns. (3 Replies)
Discussion started by: sagar_1986
3 Replies

10. Shell Programming and Scripting

Removing White spaces from a huge file

I am trying to remove whitespaces from a file containing sample data as: 457 <EOFD> Mar 1 2007 12:00:00:000AM <EOFD> Mar 31 2007 12:00:00:000AM <EOFD> system <EORD> 458 <EOFD> Mar 1 2007 12:00:00:000AM<EOFD>agf <EOFD> Apr 20 2007 9:10:56:036PM <EOFD> prodiws<EORD> . Basically these... (11 Replies)
Discussion started by: amvip
11 Replies
GNUNET-NAMESTORE(1)					      General Commands Manual					       GNUNET-NAMESTORE(1)

NAME
gnunet-namestore - manipulate GNUnet zones SYNOPSIS
gnunet-namestore [options]-zZONEFILE DESCRIPTION
gnunet-namestore can be used to create and manipulate a GNS zone. OPTIONS
-a, --add Desired operation is adding a record -c FILENAME, --config=FILENAME Use the configuration file FILENAME. -d, --delete Desired operation is deleting a record -D, --display Desired operation is listing of matching records -e TIME, --expiration=TIME Specifies expiration time of record to add; format is relative time, i.e "1 h" or "7 d 30 m". Supported units are "ms", "s", "min" or "minutes", "h" (hours), "d" (days) and "a" (years). -h, --help Print short help on options. -L LOGLEVEL, --loglevel=LOGLEVEL Use LOGLEVEL for logging. Valid values are DEBUG, INFO, WARNING and ERROR. -n NAME, --name=NAME Name of the record to add/delete/display -t TYPE, --type=TYPE Type of the record to add/delete/display (i.e. "A", "AAAA", "NS", "PKEY", "MX" etc.) -v, --version Print GNUnet version number. -V VALUE, --value=VALUE Value to store or remove from the GNS zone. Specific format depends on the record type. A records expect a dotted decimal IPv4 address, AAAA records an IPv6 address, PKEY a public key in GNUnet's printable format, and CNAME and NS records should be a domain name. -z FILENAME, --zonekey=FILENAME Specifies the filename with the private key for the zone (mandatory option) BUGS
Report bugs by using Mantis <https://gnunet.org/bugs/> or by sending electronic mail to <gnunet-developers@gnu.org> SEE ALSO
gnunet-gns(1) GNUnet Mar 5, 2012 GNUNET-NAMESTORE(1)
All times are GMT -4. The time now is 06:50 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy