04-10-2009
Issues with filtering duplicate records using gawk script
Thanks for your reply.
consider trade file is sorted based on COB date(Close of business date)
e.g
30/10/2009,tradeid1,..,..
26/10/2009,tradeId1,..,..
30/10/2009,tradeId2,..,..
26/10/2009,tradeId2,..,..
Now I want to remove duplicate records by comparing trade ids and COB dates.
If trade ids are same then i need latest COB date record.
30/10/2009,tradeid1,..,..
30/10/2009,tradeId1,..,..
also gawk script which i wrote it should work even if no duplicate records are there.
the script which i wrote works fine when no duplicate records are there but it doest not work if duplicate records are there.
10 More Discussions You Might Find Interesting
1. UNIX for Advanced & Expert Users
Does anybody know a command that filters duplicate lines out of a file. Similar to the uniq command but can handle duplicate lines no matter where they occur in a file? (9 Replies)
Discussion started by: AreaMan
9 Replies
2. UNIX for Dummies Questions & Answers
Hi all,
I would like to extract records of a file based on a condition. The file contains 47 fields, and I would like to extract only those records that match a certain value in one of the columns, e.g.
COL1 COL2 COL3 ............... COL47
1 XX 45 ... (4 Replies)
Discussion started by: risk_sly
4 Replies
3. Linux
I have huge txt file having millions of trade data.
For e.g
Trade.txt (first 8 lines in the file is header info)
COB_DATE,TRADE_ID,SOURCE_SYSTEM_TRADE_ID,TRADE_GROUP_ID,
TRADE_TYPE,DEALER_NAME,EXTERNAL_COUNTERPARTY_ID,
EXTERNAL_COUNTERPARTY_NAME,DB_COUNTERPARTY_ID,... (6 Replies)
Discussion started by: nmumbarkar
6 Replies
4. Shell Programming and Scripting
Hi,
I am new to unix and would greatly appreciate some help.
I have a file containing multiple colums containing different sets of data e.g.
File 1:
John Ireland 27_December_69
Mary England 13_March_55
Mike France 02_June_80
I am currently using the awk... (10 Replies)
Discussion started by: crunchie
10 Replies
5. Shell Programming and Scripting
Hi,
I am currently using the sed and awk commands to filter a file that has multiple sets of data in different columns. An example of part of the file i am filtering is as follows;
Sat Oct 2 07:42:45 2010 01:33:46 R1_CAR_12.34
Sun Oct 3 13:09:53 2010 00:02:34 R2_BUS_56.78
Sun... (4 Replies)
Discussion started by: crunchie
4 Replies
6. Shell Programming and Scripting
Hi,
I am currently filtering a file that has multiple sets of data. An example of some of the data is as follows;
Sat Oct 2 07:42:45 2010 01:33:46 R1_CAR_12.34
Sun Oct 3 13:09:53 2010 00:02:34 R2_BUS_56.78
Sun Oct 3 21:11:39 2010 00:43:21 R3_TRAIN_COACH_90.12
Mon Oct 4... (1 Reply)
Discussion started by: crunchie
1 Replies
7. Linux
I have log file having Traffic line
2011-05-21 15:11:50.356599 TCP (6), length: 52) 10.10.10.1.3020 > 10.10.10.254.50404:
2011-05-21 15:11:50.652739 TCP (6), length: 52) 10.10.10.254.50404 > 10.10.10.1.3020:
2011-05-21 15:11:50.652558 TCP (6), length: 89) 10.10.10.1.3020 >... (1 Reply)
Discussion started by: Rastamed
1 Replies
8. Shell Programming and Scripting
Hi I have a file like this. I need to eliminate lines with first column having the same value 10 times.
13 18 1 + chromosome 1, 122638287 AGAGTATGGTCGCGGTTG
13 18 1 + chromosome 1, 128904080 AGAGTATGGTCGCGGTTG
13 18 1 - chromosome 14, 13627938 CAACCGCGACCATACTCT
13 18 1 + chromosome 1,... (5 Replies)
Discussion started by: polsum
5 Replies
9. Shell Programming and Scripting
Hi,
I have following records in a file
more file1.txt
setting applicaction ABC for user
setting applicaction CDE for user
setting applicaction XXX for user
logging applicaction XXX for user
I need to filter out records which have strings " setting... (5 Replies)
Discussion started by: manid
5 Replies
10. UNIX for Beginners Questions & Answers
Hi,
I tried filtering the records in a csv file using "awk" command listed below.
awk -F"~" '$4 ~ /Active/{print }' inputfile > outputfile
The output always has all the entries.
The same command worked for different users from one of the forum links.
content of file I was... (3 Replies)
Discussion started by: sunilmudikonda
3 Replies