CSV file:Find duplicates, save original and duplicate records in a new file Post: 302536521

Sponsored Content

Top Forums UNIX for Dummies Questions & Answers CSV file:Find duplicates, save original and duplicate records in a new file Post 302536521 by arvindosu on Tuesday 5th of July 2011 04:06:18 PM

07-05-2011

Registered User

Thanks a ton, Corona688! I'm not sure if I was clear in my previous post. Basically, I also have files of about 15GB size. So if I take one of these files, the sample data would look like this-

72426459560 2010-06-2 ABC LC11100619758

95327GNFA4S 2010-06-2 ABC 97BCX3AMD10G

95327GNFA4S 2010-06-2 ABC 97BCX3AMKLMO

900278VGA4T 2010-06-2 ABC QVA697C8LAYMACBF

900278VG567 2010-06-2 ABC QVA697C8LAYMACBF

(column 3 would be the same for the entire 15GB file)

From this file: I want to find duplicates in column one and four. The output would look something like this:

5327GNFA4S 2010-06-2 ABC 97BCX3AMD10G

95327GNFA4S 2010-06-2 ABC 97BCX3AMKLMO

900278VGA4T 2010-06-2 ABC QVA697C8LAYMACBF

900278VG567 2010-06-2 ABC QVA697C8LAYMACBF

I can use a code that will find me duplicates only in column 1 and save it in file ABC.txt. Then rerun the same code to find duplicates in column 4 and save it in ABC2.txt. Two different files..

I want to save all the duplicates with original records (as in the example above) in a new CSV file.

Last edited by arvindosu; 07-05-2011 at 05:16 PM..

arvindosu

View Public Profile for arvindosu

Find all posts by arvindosu

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to find Duplicate Records in a text file

Hi all pls help me by providing soln for my problem I'm having a text file which contains duplicate records . Example: abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 tas 3420 3562 ...

2. Shell Programming and Scripting

find out duplicate records in file?

Dear All, I have one file which looks like : account1:passwd1 account2:passwd2 account3:passwd3 account1:passwd4 account5:passwd5 account6:passwd6 you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in...

3. Shell Programming and Scripting

Find Duplicate records in first Column in File

Hi, Need to find a duplicate records on the first column, ANU4501710430989 0000000W20389390 ANU4501710430989 0000000W67065483 ANU4501130050520 0000000W80838713 ANU4501210170685 0000000W69246611...

4. Shell Programming and Scripting

Deleting duplicate records from file 1 if records from file 2 match

I have 2 files "File 1" is delimited by ";" and "File 2" is delimited by "|". File 1 below (3 record shown): Doc1;03/01/2012;New York;6 Main Street;Mr. Smith 1;Mr. Jones Doc2;03/01/2012;Syracuse;876 Broadway;John Davis;Barbara Lull Doc3;03/01/2012;Buffalo;779 Old Windy Road;Charles...

5. Shell Programming and Scripting

FILE_ID extraction from file name and save it in CSV file after looping through each folders

FILE_ID extraction from file name and save it in CSV file after looping through each folders My files are located in UNIX Server, i want to extract file_id and file_name from each file .and save it in a CSV file. How do I do that? I have folders in unix environment, directory structure is...

6. Shell Programming and Scripting

Save output of updated csv file as csv file itself

Hi, all I want to sort a csv file based on timestamp from oldest to newest and save the output as csv file itself. Here is an example of my csv file. test.csv SourceFile,DateTimeOriginal /home/intannf/foto/IMG_0739.JPG,2015:02:17 11:32:21 /home/intannf/foto/IMG_0749.JPG,2015:02:17 11:37:28...

7. Shell Programming and Scripting

Save output of updated csv file as csv file itself, part 2

Hi, I have another problem. I want to sort another csv file by the first field. result.csv SourceFile,Airspeed,GPSLatitude,GPSLongitude,Temperature,Pressure,Altitude,Roll,Pitch,Yaw /home/intannf/foto5/2015_0313_090651_219.JPG,0.,-7.77223,110.37310,30.75,996.46,148.75,180.94,182.00,63.92 ...

8. Shell Programming and Scripting

Filter duplicate records from csv file with condition on one column

I have csv file with 30, 40 columns Pasting just three column for problem description I want to filter record if column 1 matches CN or DN then, check for values in column 2 if column contain 1235, 1235 then in column 3 values must be sequence of 2345, 2345 and if column 2 contains 6789, 6789...

9. Shell Programming and Scripting

CSV File:Filter duplicate records from column1 & another column having unique record

Hi Experts, I have csv file with 30, 40 columns Pasting just 2 column for problem description. Need to print error if below combination is not present in file check for column-1 (DocumentNumber) and filter columns where value in DocumentNumber field is same. For all such rows, the field...

LEARN ABOUT V7

macutil

MACUTIL(1)						      General Commands Manual							MACUTIL(1)

NAME

       macutil - A package that deals with MacIntosh files on a Unix system

DESCRIPTION

       macutil	is  a package that contains a number of utilities that deal with MacIntosh files on a Unix system.  It contains the following pro-
       grams:

       binhex Convert files to BinHex 4.0 compatible hexified form.

       frommac
	      Receives files from the MacIntosh on the Unix system.

       hexbin Convert hexified files to their MacIntosh format.

       macsave
	      Save a series of files from a MacBinary stream as individual files.

       macstream
	      Combine a series of files to a MacBinary stream.

       macunpack
	      Unpack a MacIntosh archive into its constituents.

       tomac  Transmits files from the Unix system to a MacIntosh.

BUGS

       This manual page is hopelessly incomplete!

SEE ALSO

       binhex(1), frommac(1), hexbin(1), macsave(1), macstream(1), macunpack(1), tomac(1)

AUTHOR

       Dik T. Winter, CWI, Amsterdam, The Netherlands (dik@cwi.nl)

3rd Berkeley Distribution					 October 22, 1992							MACUTIL(1)