Removing duplicates from delimited file based on 2 columns
Hi guys,Got a bit of a bind I'm in. I'm looking to remove duplicates from a pipe delimited file, but do so based on 2 columns. Sounds easy enough, but here's the kicker...
Column #1 is a simple ID, which is used to identify the duplicate.
Once dups are identified, I need to only keep the one with the latest date, which is column #4, in mm/dd/yyyy format. Of course, rows that don't have dup's would remain as-is.
Example input.txt:
In that scenario, the output would be rows 1, 3, and 4, since rows 2 and 5 are duplicates based on the ID and are older than the one in row 4 based on date.
The other kicker is...
The file I'm doing this with is 400,000 rows. So, I need the method to be extremely efficient and as quick as possible. I can't afford for this to take hours.
This is running on a Windows machine with GnuWin utils, as one last note.
I am definitely not enough of an expert to make this work, especially efficiently, so I'm hoping someone can help. Many thanks in advance.
Last edited by Don Cragun; 08-13-2014 at 12:29 AM..
Reason: Remove FONT tags; add CODE and ICODE tags.
HI I am having a file like this
1234
12345678
1234567890123
4321
43215678
432156789028433435
I want to get ouput as
1234567890123
432156789028433435
based on key position 1-4
I am using ksh can anyone give me an idea
Thanks
pukars (1 Reply)
I am trying to figure out how to scan a file like so:
1 ralphs office","555-555-5555","ralph@mail.com","www.ralph.com
2 margies office","555-555-5555","ralph@mail.com","www.ralph.com
3 kims office","555-555-5555","kims@mail.com","www.ralph.com
4 tims... (17 Replies)
Hi All,
I needs to fetch unique records based on a keycolumn(ie., first column1) and also I needs to get the records which are having max value on column2 in sorted manner... and duplicates have to store in another output file.
Input :
Input.txt
1234,0,x
1234,1,y
5678,10,z
9999,10,k... (7 Replies)
Hi,
I am unable to search the duplicates in a file based on the 1st,2nd,4th,5th columns in a file and also remove the duplicates in the same file.
Source filename: Filename.csv
"1","ccc","information","5000","temp","concept","new"
"1","ddd","information","6000","temp","concept","new"... (2 Replies)
Hi,
I have the input file with the below data:
12345|12|34
12345|13|23
3456|12|90
15670|12|13
12345|10|14
3456|12|13
I need to remove the duplicates based on the first field only.
I need the output like:
12345|12|34
3456|12|90
15670|12|13
The first field needs to be unique . (4 Replies)
Hi team,
I have 20 columns csv files. i want to find the duplicates in that file based on the column1 column10 column4 column6 coulnn8 coulunm2 . if those columns have same values . then it should be a duplicate record.
can one help me on finding the duplicates,
Thanks in advance.
... (2 Replies)
Hi All ,
I have a requirement where I need to remove duplicates from a fixed width file which has multiple key columns .Also , need to capture the duplicate records into another file .
File has 8 columns.
Key columns are col1 and col2.
Col1 has the length of 8 col 2 has the length of 3.
... (5 Replies)
Hi some one please help me to remove duplicates from a pipe delimited file based on first two columns.
123|asdf|sfsd|qwrer
431|yui|qwer|opws
123|asdf|pol|njio
Here My first record and last record are duplicates.As per my requirement I want all the latest records into one file.
I want the... (12 Replies)
Hello !
I'm quite new to linux but haven't found a script to do this task, unfortunately my knowledge is quite limited on shellscripts...
Could you guys help me removing the duplicate lines of a file, based only on a single "column"?
For example:
M202034357;01/2008;J30RJ021;Ciclo 01... (4 Replies)
I have /tmp dir with filename as:
010020001_S-FOR-Sort-SYEXC_20160229_2212101.marker
010020001_S-FOR-Sort-SYEXC_20160229_2212102.marker
010020001-S-XOR-Sort-SYEXC_20160229_2212104.marker
010020001-S-XOR-Sort-SYEXC_20160229_2212105.marker
010020001_S-ZOR-Sort-SYEXC_20160229_2212106.marker... (4 Replies)
Discussion started by: gnnsprapa
4 Replies
LEARN ABOUT CENTOS
tk::table
Table(3) User Contributed Perl Documentation Table(3)NAME
Tk::Table - Scrollable 2 dimensional table of Tk widgets
SYNOPSIS
use Tk::Table;
$table = $parent->Table(-rows => number,
-columns => number,
-scrollbars => anchor,
-fixedrows => number,
-fixedcolumns => number,
-takefocus => boolean);
$widget = $table->Button(...);
$old = $table->put($row,$col,$widget);
$old = $table->put($row,$col,"Text"); # simple Label
$widget = $table->get($row,$col);
$cols = $table->totalColumns;
$rows = $table->totalRows;
$table->see($widget);
$table->see($row,$col);
($row,$col) = $table->Posn($widget);
DESCRIPTION
Tk::Table is an all-perl widget/geometry manager which allows a two dimensional table of arbitary perl/Tk widgets to be displayed.
Entries in the Table are simply ordinary perl/Tk widgets. They should be created with the Table as their parent. Widgets are positioned in
the table using:
$table->put($row,$col,$widget)
If $widget is not a reference it is treated as a string, and a Lable widget is created with the string as its text.
All the widgets in each column are set to the same width - the requested width of the widest widget in the column. Likewise, all the
widgets in each row are set to the same height - the requested height of the tallest widget in the column.
A number of rows and/or columns can be marked as 'fixed' - and so can serve as 'headings' for the remainder the rows which are scrollable.
The requested size of the table as a whole is such that the number of rows specified by -rows (default 10), and number of columns specified
by -columns (default 10) can be displayed.
If the Table is told it can take the keyboard focus then cursor and scroll keys scroll the displayed widgets.
The Table will create and manage its own scrollbars if requested via -scrollbars.
The table can be emptied using
$table->clear
the widgets which were in the table are destroyed.
The Tk::Table widget is derived from a Tk::Frame, so inherits all its configure options.
BUGS / Snags / Possible enhancements
o Very large Tables consume a lot of X windows.
o No equivalent of pack's -anchor/-pad etc. options
SEE ALSO
Tk::grid, Tk::HList, Tk::TableMatrix, Tk::MListbox, Tk::Columns
perl v5.16.3 2014-06-10 Table(3)