Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Issues with filtering duplicate records using gawk script
Top Forums Shell Programming and Scripting Issues with filtering duplicate records using gawk script Post 302305860 by nmumbarkar on Friday 10th of April 2009 01:04:23 AM
Old 04-10-2009
Issues with filtering duplicate records using gawk script
Thanks for your reply.
consider trade file is sorted based on COB date(Close of business date)
e.g
30/10/2009,tradeid1,..,..
26/10/2009,tradeId1,..,..
30/10/2009,tradeId2,..,..
26/10/2009,tradeId2,..,..

Now I want to remove duplicate records by comparing trade ids and COB dates.
If trade ids are same then i need latest COB date record.
30/10/2009,tradeid1,..,..
30/10/2009,tradeId1,..,..

also gawk script which i wrote it should work even if no duplicate records are there.

the script which i wrote works fine when no duplicate records are there but it doest not work if duplicate records are there.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Filtering duplicate lines

Does anybody know a command that filters duplicate lines out of a file. Similar to the uniq command but can handle duplicate lines no matter where they occur in a file? (9 Replies)
Discussion started by: AreaMan
9 Replies

2. UNIX for Dummies Questions & Answers

Filtering records of a file based on a value of a column

Hi all, I would like to extract records of a file based on a condition. The file contains 47 fields, and I would like to extract only those records that match a certain value in one of the columns, e.g. COL1 COL2 COL3 ............... COL47 1 XX 45 ... (4 Replies)
Discussion started by: risk_sly
4 Replies

3. Linux

Need awk script for removing duplicate records

I have huge txt file having millions of trade data. For e.g Trade.txt (first 8 lines in the file is header info) COB_DATE,TRADE_ID,SOURCE_SYSTEM_TRADE_ID,TRADE_GROUP_ID, TRADE_TYPE,DEALER_NAME,EXTERNAL_COUNTERPARTY_ID, EXTERNAL_COUNTERPARTY_NAME,DB_COUNTERPARTY_ID,... (6 Replies)
Discussion started by: nmumbarkar
6 Replies

4. Shell Programming and Scripting

Filtering issues with multiple columns in a single file

Hi, I am new to unix and would greatly appreciate some help. I have a file containing multiple colums containing different sets of data e.g. File 1: John Ireland 27_December_69 Mary England 13_March_55 Mike France 02_June_80 I am currently using the awk... (10 Replies)
Discussion started by: crunchie
10 Replies

5. Shell Programming and Scripting

Filtering Issues Using sed and awk

Hi, I am currently using the sed and awk commands to filter a file that has multiple sets of data in different columns. An example of part of the file i am filtering is as follows; Sat Oct 2 07:42:45 2010 01:33:46 R1_CAR_12.34 Sun Oct 3 13:09:53 2010 00:02:34 R2_BUS_56.78 Sun... (4 Replies)
Discussion started by: crunchie
4 Replies

6. Shell Programming and Scripting

Filtering issues while using nawk

Hi, I am currently filtering a file that has multiple sets of data. An example of some of the data is as follows; Sat Oct 2 07:42:45 2010 01:33:46 R1_CAR_12.34 Sun Oct 3 13:09:53 2010 00:02:34 R2_BUS_56.78 Sun Oct 3 21:11:39 2010 00:43:21 R3_TRAIN_COACH_90.12 Mon Oct 4... (1 Reply)
Discussion started by: crunchie
1 Replies

7. Linux

Need awk script for removing duplicate records

I have log file having Traffic line 2011-05-21 15:11:50.356599 TCP (6), length: 52) 10.10.10.1.3020 > 10.10.10.254.50404: 2011-05-21 15:11:50.652739 TCP (6), length: 52) 10.10.10.254.50404 > 10.10.10.1.3020: 2011-05-21 15:11:50.652558 TCP (6), length: 89) 10.10.10.1.3020 >... (1 Reply)
Discussion started by: Rastamed
1 Replies

8. Shell Programming and Scripting

Perl: filtering lines based on duplicate values in a column

Hi I have a file like this. I need to eliminate lines with first column having the same value 10 times. 13 18 1 + chromosome 1, 122638287 AGAGTATGGTCGCGGTTG 13 18 1 + chromosome 1, 128904080 AGAGTATGGTCGCGGTTG 13 18 1 - chromosome 14, 13627938 CAACCGCGACCATACTCT 13 18 1 + chromosome 1,... (5 Replies)
Discussion started by: polsum
5 Replies

9. Shell Programming and Scripting

Need help with filtering records in a file

Hi, I have following records in a file more file1.txt setting applicaction ABC for user setting applicaction CDE for user setting applicaction XXX for user logging applicaction XXX for user I need to filter out records which have strings " setting... (5 Replies)
Discussion started by: manid
5 Replies

10. UNIX for Beginners Questions & Answers

Filtering records of a csv file based on a value of a column

Hi, I tried filtering the records in a csv file using "awk" command listed below. awk -F"~" '$4 ~ /Active/{print }' inputfile > outputfile The output always has all the entries. The same command worked for different users from one of the forum links. content of file I was... (3 Replies)
Discussion started by: sunilmudikonda
3 Replies

LEARN ABOUT DEBIAN

fl-record

FL-RECORD(1)							   Debian manual						      FL-RECORD(1)

NAME

       fl-record - records a live funkload test using TCPWatch

SYNOPSIS

       fl-record [ options ] test_case

DESCRIPTION

       fl-record uses TCPWatch to setup a proxy server which is then used by a browser and records activities. It generates a Funkload script or a
       unit test if a name is passed to the script.

OPTIONS

       --version
	      Show program's version number and exit

       --help, -h
	      Show a help message and exit

       --verbose, -v
	      Verbose output

       --port=PORT, -p PORT
	      TCP port where the proxy listens (Defaults to 8090)

       --tcp-watch-input=TCPWATCH_PATH, -i TCPWATCH_PATH
	      Path to an existing TCPWatch capture

       --loop=LOOP, -l LOOP
	      Loop mode

SEE ALSO

       fl-build-report(1), fl-credential-ctl(1), fl-install-demo(1), fl-monitor-ctl(1), fl-run-bench(1), fl-run-test(1).

AUTHOR

       Funkload was written by Benoit Delbosc.

       This manual page was written by Jose Parrella <bureado@debian.org>, for the Debian project (but may be used by others).

Debian Project							   26 March 2009						      FL-RECORD(1)
All times are GMT -4. The time now is 09:07 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy