Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: CSV file:Find duplicates, save original and duplicate records in a new file
Top Forums UNIX for Dummies Questions & Answers CSV file:Find duplicates, save original and duplicate records in a new file Post 302536514 by Corona688 on Tuesday 5th of July 2011 03:52:57 PM
Old 07-05-2011
15 gigs is still too large to do a sort on.

Here's a smaller-scale version that gets column 3. I set the split size to 100K so I could do sensible testing on my 5 meg test data.

Code:
#!/bin/sh

COL=3

# Break file into sortable chunks xaa - xzz
# Try between 500M and 2G for your gigantic file.
split -C 100K < megadata.txt

# Sort each individual file on given column
for FILE in x??
do
        sort -k $COL < "${FILE}" > "${FILE}.tmp"
        rm -f "${FILE}"
        mv "${FILE}.tmp" "${FILE}"
        shift
done

# Merge all the sorted files into one great big list.
# run it through awk, which will print only the first row which has any
# particular value for column COL.
sort -k $COL -m x?? | awk -v COL=${COL} '{
        if(LAST != $COL)        print;
        LAST=$COL;      }' > megasorted.txt

# Delete splitted files
# rm -f x??

It will take only the first row to have a particular value for column COL and no others.
Last edited by Corona688; 07-05-2011 at 05:00 PM..
This User Gave Thanks to Corona688 For This Post:
arvindosu
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to find Duplicate Records in a text file

Hi all pls help me by providing soln for my problem I'm having a text file which contains duplicate records . Example: abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 tas 3420 3562 ... (1 Reply)
Discussion started by: G.Aavudai
1 Replies

2. Shell Programming and Scripting

find out duplicate records in file?

Dear All, I have one file which looks like : account1:passwd1 account2:passwd2 account3:passwd3 account1:passwd4 account5:passwd5 account6:passwd6 you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in... (3 Replies)
Discussion started by: tiger2000
3 Replies

3. Shell Programming and Scripting

Find Duplicate records in first Column in File

Hi, Need to find a duplicate records on the first column, ANU4501710430989 0000000W20389390 ANU4501710430989 0000000W67065483 ANU4501130050520 0000000W80838713 ANU4501210170685 0000000W69246611... (3 Replies)
Discussion started by: Murugesh
3 Replies

4. Shell Programming and Scripting

Deleting duplicate records from file 1 if records from file 2 match

I have 2 files "File 1" is delimited by ";" and "File 2" is delimited by "|". File 1 below (3 record shown): Doc1;03/01/2012;New York;6 Main Street;Mr. Smith 1;Mr. Jones Doc2;03/01/2012;Syracuse;876 Broadway;John Davis;Barbara Lull Doc3;03/01/2012;Buffalo;779 Old Windy Road;Charles... (2 Replies)
Discussion started by: vestport
2 Replies

5. Shell Programming and Scripting

FILE_ID extraction from file name and save it in CSV file after looping through each folders

FILE_ID extraction from file name and save it in CSV file after looping through each folders My files are located in UNIX Server, i want to extract file_id and file_name from each file .and save it in a CSV file. How do I do that? I have folders in unix environment, directory structure is... (15 Replies)
Discussion started by: princetd001
15 Replies

6. Shell Programming and Scripting

Save output of updated csv file as csv file itself

Hi, all I want to sort a csv file based on timestamp from oldest to newest and save the output as csv file itself. Here is an example of my csv file. test.csv SourceFile,DateTimeOriginal /home/intannf/foto/IMG_0739.JPG,2015:02:17 11:32:21 /home/intannf/foto/IMG_0749.JPG,2015:02:17 11:37:28... (10 Replies)
Discussion started by: refrain
10 Replies

7. Shell Programming and Scripting

Save output of updated csv file as csv file itself, part 2

Hi, I have another problem. I want to sort another csv file by the first field. result.csv SourceFile,Airspeed,GPSLatitude,GPSLongitude,Temperature,Pressure,Altitude,Roll,Pitch,Yaw /home/intannf/foto5/2015_0313_090651_219.JPG,0.,-7.77223,110.37310,30.75,996.46,148.75,180.94,182.00,63.92 ... (2 Replies)
Discussion started by: refrain
2 Replies

8. Shell Programming and Scripting

Filter duplicate records from csv file with condition on one column

I have csv file with 30, 40 columns Pasting just three column for problem description I want to filter record if column 1 matches CN or DN then, check for values in column 2 if column contain 1235, 1235 then in column 3 values must be sequence of 2345, 2345 and if column 2 contains 6789, 6789... (5 Replies)
Discussion started by: as7951
5 Replies

9. Shell Programming and Scripting

CSV File:Filter duplicate records from column1 & another column having unique record

Hi Experts, I have csv file with 30, 40 columns Pasting just 2 column for problem description. Need to print error if below combination is not present in file check for column-1 (DocumentNumber) and filter columns where value in DocumentNumber field is same. For all such rows, the field... (7 Replies)
Discussion started by: as7951
7 Replies

LEARN ABOUT DEBIAN

rdf::trine::exporter::csv

RDF::Trine::Exporter::CSV(3pm)				User Contributed Perl Documentation			    RDF::Trine::Exporter::CSV(3pm)

NAME

       RDF::Trine::Exporter::CSV - Export RDF data to CSV

VERSION

       This document describes RDF::Trine::Exporter::CSV version 1.000

SYNOPSIS

	use RDF::Trine::Exporter::CSV;

DESCRIPTION

       The RDF::Trine::Exporter::CSV class provides an API for serializing RDF data to CSV strings and files.

METHODS

       "new ( sep_char => $sep_char, quote => $bool )"
	   Returns a new RDF::Trine::Exporter::CSV object. If $sep_char is provided, it is used as the separator character in CSV serialization,
	   otherwise a comma (",") is used.

       "serialize_iterator_to_file ( $file, $iterator )"
	   Serializes the bindings objects produced by $iterator, printing the results to the supplied filehandle "<$fh">.

       "serialize_iterator_to_string ( $iterator )"
	   Serializes the bindings objects produced by $iterator, returning the result as a string.

BUGS

       Please report any bugs or feature requests to through the GitHub web interface at <https://github.com/kasei/perlrdf/issues>.

AUTHOR

       Gregory Todd Williams  "<gwilliams@cpan.org>"

COPYRIGHT

       Copyright (c) 2006-2012 Gregory Todd Williams. This program is free software; you can redistribute it and/or modify it under the same terms
       as Perl itself.

perl v5.14.2							    2012-06-29					    RDF::Trine::Exporter::CSV(3pm)
All times are GMT -4. The time now is 10:26 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy