Sponsored Content
Full Discussion: Checking file for duplicates
Top Forums Shell Programming and Scripting Checking file for duplicates Post 302422741 by methyl on Wednesday 19th of May 2010 09:28:47 AM
Old 05-19-2010
We must assume you have a mainstream database engine which can handle CSV files and a computer which has capacity for this task. You really don't mention much about the data or the computer.

At a design level, each record must contain a unique key and whatever information is a parameter to the "purge". Unless you know the source "purge" rules your database of "data already processed" will just grow.


It would make more sense to fix the data feed design at at source. A convention is to mark the record in the source database with a unique extract run reference to prevent repeat extracts - whilst also allowing a rerun.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Avoid Duplicates in a file

Hi Gurus, I had a question regarding avoiding duplicates.i have a file abc.txt abc.txt ------- READER_1_1_1> HIER_28056 XML Reader: Error occurred while parsing:; line number ; column number READER_1_3_1> Sun Mar 23 23:52:48 2008 READER_1_3_1> HIER_28056 XML Reader: Error occurred while... (7 Replies)
Discussion started by: pssandeep
7 Replies

2. Shell Programming and Scripting

Remove duplicates from a file

Hi, I need to remove duplicates from a file. The file will be like this 0003 10101 20100120 abcdefghi 0003 10101 20100121 abcdefghi 0003 10101 20100122 abcdefghi 0003 10102 20100120 abcdefghi 0003 10103 20100120 abcdefghi 0003 10103 20100121 abcdefghi Here if the first colum and... (6 Replies)
Discussion started by: gpaulose
6 Replies

3. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

4. Shell Programming and Scripting

Removing Duplicates from file

Hi Experts, Please check the following new requirement. I got data like the following in a file. FILE_HEADER 01cbbfde7898410| 3477945| home| 1 01cbc275d2c122| 3478234| WORK| 1 01cbbe4362743da| 3496386| Rich Spare| 1 01cbc275d2c122| 3478234| WORK| 1 This is pipe separated file with... (3 Replies)
Discussion started by: tinufarid
3 Replies

5. Shell Programming and Scripting

Duplicates in an XML file

Hi All, I have an xml file that contains information like this <ID>574922<COMMENT>TEXT TEXT TEXT</COMMENT></ID> <ID>574922<COMMENT>TEXT TEXT TEXT</COMMENT></ID> <ID>412659<COMMENT>TEXT TEXT TEXT TEXT TEXT</COMMENT></ID> <ID>873520<COMMENT>TEXT</COMMENT></ID>... (5 Replies)
Discussion started by: TasosARISFC
5 Replies

6. Shell Programming and Scripting

Remove the partial duplicates by checking the length of a field

Hi Folks - I'm quite new to awk and didn't come across such issues before. The problem statement is that, I've a file with duplicate records in 3rd and 4th fields. The sample is as below: aaaaaa|a12|45|56 abbbbaaa|a12|45|56 bbaabb|b1|51|45 bbbbbabbb|b2|51|45 aaabbbaaaa|a11|45|56 ... (3 Replies)
Discussion started by: asyed
3 Replies

7. Programming

[Solved] Removing duplicates from the file and saving as new file

Dear All I have 200 data files and each files has many duplicates. I am looking for the automated awk script such that it checks and removes the duplicates from the each file and saving them as new files for all 200 files in the respective folder. For example my data looks like this.. ... (12 Replies)
Discussion started by: bala06
12 Replies

8. UNIX for Dummies Questions & Answers

Remove duplicates from a file

Can u tell me how to remove duplicate records from a file? (11 Replies)
Discussion started by: saga20
11 Replies

9. UNIX for Dummies Questions & Answers

Removing duplicates from a file

Hi All, I am merging files coming from 2 different systems ,while doing that I am getting duplicates entries in the merged file I,01,000131,764,2,4.00 I,01,000131,765,2,4.00 I,01,000131,772,2,4.00 I,01,000131,773,2,4.00 I,01,000168,762,2,2.00 I,01,000168,763,2,2.00... (5 Replies)
Discussion started by: Sri3001
5 Replies

10. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3.I have tried previous post also,but in that complete line must be similar.In this case i have to verify first column only regardless what is the content in succeeding columns. (3 Replies)
Discussion started by: sagar_1986
3 Replies
ADPLUGDB(1)							   User Commands						       ADPLUGDB(1)

NAME
adplugdb - AdPlug database maintenance utility SYNOPSIS
adplugdb [OPTION]... COMMAND [ARGUMENT]... DESCRIPTION
adplugdb maintains database files in AdPlug database format. It can add, list and remove records within a central database, or merge a set of databases together into one single database. adplugdb always operates on a central database file. The location of this database file is determined by first checking if the user has a home directory. If a home directory is present, the database file will be located in ~/.adplug/adplug.db. If a home directory is not present, a database file adplug.db will be looked for in the current working directory. A system-wide database file can be used instead, by specifying the -s commandline option. The system-wide database file is located in /usr/com/adplug/adplug.db and may only be manipulated as the superuser. An arbitrary database file might be used as well, by specifying the -d commandline parameter. Only one database file may be manipulated at a time. EXIT STATUS
adplugdb returns with a successful exit status (0 on most systems) on successful operation. An unsuccessful exit status (1 on most systems) is returned otherwise. COMMANDS
Commands control the main operation mode of adplugdb. Commands can have a number of arguments. Only one command may be specified at a time. add This command takes a list of filenames, separated by spaces, as arguments. Each file is examined and a record is added to the data- base if the file is supported by AdPlug. By default, the record will be of type Plain, unless the -t commandline option is specified (see below). The default comment entry is the specified filename. If a record for a file is already in the database, it will be replaced by the new record. list This command takes an optional list of filenames or keys, separated by spaces, as arguments. Each file is examined and the corre- sponding record is looked up from the database and displayed on stdout, in a human-readable form. If no arguments are given, all records from the database are displayed. remove This command takes a list of filenames or keys, separated by spaces, as arguments. Each file is examined and the corresponding record is removed from the database. merge This command takes a list of database filenames, separated by spaces, as arguments. Each database file is loaded and the contents are merged and written to the central database file. The database files are processed in the order they are specified on the comman- dline. Records from databases that were specified earlier take precedence over records from databases that were specified later. Records from the central database take precedence over all other records. This means that only additional records from the other databases will be added to the central database and if a record is found that is not already in the central database, the version from the earliest specified database that contains this record will be taken. In no way will records ever be overwritten in the cen- tral database. OPTIONS
The order of the option commandline parameters is not important. Database options: -d <file> Specify an arbitrary file to use as the central database. -s Use the system-wide database file as the central database. This option is only present if adplugdb was compiled with system-wide database file support. -t <type> Specify a record type to be used as the type for all newly added records. Each record needs a special type to be useful to AdPlug's players. The commandline help, displayed using the -h commandline option, presents a list of types that may be specified. -c Prompt for record comment. If this option is given, the user will be prompted and asked for each newly added record's comment. -k Specify keys instead of filenames. If this option is given, all command arguments that normally are filenames are expected to be record keys instead. Each record in the database has a unique identifying key, generated from the corresponding file's contents. To manipulate a record entry, you either must have the exact same file and specify its name, or you specify the record's key, using this option. Keys are specified the same way they are displayed using the list command, as CRC16:CRC32 value in hexadecimal format. Generic options: -q, --quiet Be more quiet. -v, --verbose Be more verbose. -h, --help Show summary of commandline commands, arguments and options. -V, --version Show version and author information of the program. AUTHOR
Simon Peter <dn.tlp@gmx.net> AdPlug database maintenance utility 2.2.1 March 4, 2006 ADPLUGDB(1)
All times are GMT -4. The time now is 03:19 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy