01-12-2007
How to extract duplicate records with associated header record
All,
I have a task to search through several hundred files and extract duplicate detail records and keep them grouped with their header record. If no duplicate detail record exists, don't pull the header. For example, an input file could look like this:
input.txt
HA
D1
D2
D2
D3
D4
D4
HB
D1
D2
HC
D1
D1
D2
D3
D3
The output would be:
output.txt
HA
D2
D4
HC
D1
D3
Would it be possible to do this with AWK? I do not know python.
Thank you for your time.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I have a flat file and need to count no of records in the file less the header and the trailer record.
I would appreciate any and all asistance
Thanks
Hadi Lalani (2 Replies)
Discussion started by: guiguy
2 Replies
2. UNIX for Dummies Questions & Answers
Hello,
after 9 months of archiving 1000 files,
now, i need to change the order of fields in the header record.
some very large, space padded files.
HEADERCAS05212008D0210DOMEST01(spacepadded to record length 210)
must now be
05212008HEADERCASD0210DOMEST01(spacepadded to record length 210)
... (1 Reply)
Discussion started by: JohnMario
1 Replies
3. Shell Programming and Scripting
A header record is to be inserted in the begining of a flat file without using extra file or new file. It should be inserted into same file. Advace thanks for all help... (7 Replies)
Discussion started by: shreekrishnagd
7 Replies
4. Shell Programming and Scripting
Hi All,
I have got a requirement.
I have a source file, EMPFULL.txt and I need to split the data for every 30 records
and place a Typical Header as below with system and page number too.
2012.01.03 Employee Dept Report 1... (6 Replies)
Discussion started by: srk409
6 Replies
5. Shell Programming and Scripting
I have 2 files
"File 1" is delimited by ";" and "File 2" is delimited by "|".
File 1 below (3 record shown):
Doc1;03/01/2012;New York;6 Main Street;Mr. Smith 1;Mr. Jones
Doc2;03/01/2012;Syracuse;876 Broadway;John Davis;Barbara Lull
Doc3;03/01/2012;Buffalo;779 Old Windy Road;Charles... (2 Replies)
Discussion started by: vestport
2 Replies
6. Shell Programming and Scripting
All,
I currently have a requirement to fetch a Date value from a table. And then insert a Header record into a file along with that date value.
ex:
echo "HDR"" "`date +%Y%j` `date +%Y%m%d`
In the above example I used julian date and standard date using Current Date. But the requirement... (0 Replies)
Discussion started by: cmaroju
0 Replies
7. Shell Programming and Scripting
I'm using a shell script to manipulate a data file. I have a large file with two sets of data samples (tracking memory consumption) taken over a long period of time, so I have many samples. The problem is that all the data is in the same file so that each sample contains two sets of data.... (2 Replies)
Discussion started by: abercrom
2 Replies
8. Shell Programming and Scripting
I have test.xml
<emp><id>101</id><name>AAA</name><date>06/06/14 1811</date></emp>
<Join><id>101</id><city>london</city><date>06/06/14 2011</date></join>
<Join><id>101</id><city>new york</city><date>06/06/14 1811</date></join>
<Join><id>101</id><city>sydney</city><date>06/06/14... (2 Replies)
Discussion started by: vsraju
2 Replies
9. UNIX for Beginners Questions & Answers
Hi,
I have a backup report that unfortunately has some kind of hanging indent thing where the first line contains one column more than the others
I managed to get the output that I wanted using awk, but just wanting to know if there is short way of doing it using the same awk
Below is what... (2 Replies)
Discussion started by: newbie_01
2 Replies
10. Shell Programming and Scripting
Hi Experts,
I have csv file with 30, 40 columns
Pasting just 2 column for problem description.
Need to print error if below combination is not present in file
check for column-1 (DocumentNumber) and filter columns where value in DocumentNumber field is same.
For all such rows, the field... (7 Replies)
Discussion started by: as7951
7 Replies
RAGATOR(1) General Commands Manual RAGATOR(1)
NAME
ragator - aggregate argus(8) data file entries.
COPYRIGHT
Copyright (c) 2000-2003 QoSient. All rights reserved.
SYNOPSIS
ragator [-f ragator.conf] [raoptions]
DESCRIPTION
Ragator reads argus(8) data from an argus-file, and merges matching argus flow activity records together. In its default mode of options,
this effectively converts argus(8) files from detail to non-detail mode, and merges periodic flow report records to a single argus record,
thus compressing the argus(8) file to a reduced size.
You can modify the aggregation strategy used by ragator to merge records together, by using the -f ragator.conf option. See ragator(5) for
a complete description of the format and syntax of the flow model file.
OPTIONS
Ragator, like all ra based clients, supports a number of ra options including filtering of input argus records through a terminating filter
expression. See ra(1) for a complete description of ra options.
AUTHORS
Carter Bullard (carter@qosient.com).
SEE ALSO
ragator(5) ra(1), rarc(5), argus(8) tcpdump(1),
21 July 1995 RAGATOR(1)