Sponsored Content
Top Forums Shell Programming and Scripting remove duplicated xml record in a file under unix Post 302089631 by happyv on Wednesday 20th of September 2006 06:34:51 AM
Old 09-20-2006
Is the Perl can run under ksh Unix?

Also, the record is a bit difference...it look like

record1:
this is testing
my id is 2001
end:
record2:
this is testing2
my id is 2002
end:
record3:
this is testing
my id is 2002
end:
record4:
this is testing2
my id is 2002
end:

For the above, record 2 and 4 is duplicated. Because of the "id" and "testing2" is the same. if only one line is the same which is not called duplicated..

So Perl or any friend can help for the script?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove duplicated columns

hi all, i have a file contain multicolumns, this file is sorted by col2 and col3. i want to remove the duplicated columns if the col2 and col3 are the same in another line. example fileA AA BB CC DD CC XX CC DD BB CC ZZ FF DD FF HH HH the output is AA BB CC DD BB CC ZZ FF... (6 Replies)
Discussion started by: kamel.seg
6 Replies

2. UNIX for Dummies Questions & Answers

how to read record by record from a file in unix

Hi guys, i have a big file with the following format.This includes header(H),detail(D) and trailer(T) information in the file.My problem is i have to search for the character "6h" at 14 th and 15 th position in all the records .if it is there i have to write all those records into a... (1 Reply)
Discussion started by: raoscb
1 Replies

3. UNIX for Advanced & Expert Users

How to read an Xml record contained in a file--urgent

Hi I have an xml file which has multiple xml records.. I don't know how to read those records and pipe them to another shell command the file is like <abc>z<def>y<ghi>x........</ghi></def></abc> (1st record) <jkl>z<mno>y<pqr>x........</pqr></mno></jkl> (2nd record) Each record end... (4 Replies)
Discussion started by: aixjadoo
4 Replies

4. Shell Programming and Scripting

How to remove xml namespace from xml file using shell script?

I have an xml file: <AutoData xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <Table1> <Data1 10 </Data1> <Data2 20 </Data2> <Data3 40 </Data3> <Table1> </AutoData> and I have to remove the portion xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" only. I tried using sed... (10 Replies)
Discussion started by: Gary1978
10 Replies

5. Shell Programming and Scripting

Help with remove duplicated content

Input file: hcmv-US25-2-3p hsa-3160-5 hcmv-US33 hsa-47 hcmv-UL70-3p hsa-4508 hcmv-UL70-3p hsa-4486 hcms-US25 hsa-360-5 hcms-US25 hsa-4 hcms-US25 hsa-458 hcms-US25 hsa-44812 . . Desired Output file: hcmv-US25-2-3p hsa-3160-5 hcmv-US33 hsa-47 hcmv-UL70-3p hsa-4508 hsa-4486... (3 Replies)
Discussion started by: perl_beginner
3 Replies

6. UNIX for Dummies Questions & Answers

Delete a record in a xml file using shell scripting

find pattern, delete line with pattern and 3 lines above and 8 lines below the pattern. The pattern is "isup". The entire record with starting tag <record> and ending tag </record> containing the pattern is to be deleted and the rest to be retained. <record> ... (4 Replies)
Discussion started by: sdesstp
4 Replies

7. Shell Programming and Scripting

How to remove duplicated lines?

Hi, if i have a file like this: Query=1 a a b c c c d Query=2 b b b c c e . . . (7 Replies)
Discussion started by: the_simpsons
7 Replies

8. Shell Programming and Scripting

Extract timestamp from first record in xml file and it checks if not it will replace first record

I have test.xml <emp><id>101</id><name>AAA</name><date>06/06/14 1811</date></emp> <Join><id>101</id><city>london</city><date>06/06/14 2011</date></join> <Join><id>101</id><city>new york</city><date>06/06/14 1811</date></join> <Join><id>101</id><city>sydney</city><date>06/06/14... (2 Replies)
Discussion started by: vsraju
2 Replies

9. Shell Programming and Scripting

How to remove duplicated column in a text file?

Dear all, How can I remove duplicated column in a text file? Input: LG10_PM_map_19_LEnd 1000560 G AA AA AA AA AA GG LG10_PM_map_19_LEnd 1005621 G GG GG GG AA AA GG LG10_PM_map_19_LEnd 1011214 A AA AA AA AA GG GG LG10_PM_map_19_LEnd 1011673 T TT TT TT TT CC CC... (1 Reply)
Discussion started by: huiyee1
1 Replies

10. Shell Programming and Scripting

Remove duplicated records and update last line record counts

Hi Gurus, I need to remove duplicate line in file and update TRAILER (last line) record count. the file is comma delimited, field 2 is key to identify duplicated record. I can use below command to remove duplicated. but don't know how to replace last line 2nd field to new count. awk -F","... (11 Replies)
Discussion started by: green_k
11 Replies
XML::Filter::Sort::BufferMgr(3pm)			User Contributed Perl Documentation			 XML::Filter::Sort::BufferMgr(3pm)

NAME
XML::Filter::Sort::BufferMgr - Implementation class used by XML::Filter::Sort DESCRIPTION
The documentation is targetted at developers wishing to extend or replace this class. For user documentation, see XML::Filter::Sort. Two classes are used to implement buffering records and spooling them back out in sorted order as SAX events. One instance of the XML::Filter::Sort::Buffer class is used to buffer each record and one or more instances of the XML::Filter::Sort::BufferMgr class are used to manage the buffers. API METHODS
The API of this module as used by XML::Filter::Sort::Buffer consists of the following sequence of method calls: 1. When the first 'record' in a sequence is encountered, XML::Filter::Sort creates a XML::Filter::Sort::BufferMgr object using the "new()" method. 2. XML::Filter::Sort calls the buffer manager's "new_buffer()" method to get a XML::Filter::Sort::Buffer object and all SAX events are directed to this object until the end of the record is encountered. The following events are supported by the current buffer implementation: start_element() characters() comment() processing_instruction() end_element() 3. When the end of the record is detected, XML::Filter::Sort calls the buffer manager's "close_buffer()" method, which in turn calls the buffer's "close()" method. The "close()" method returns a list of values for the sort keys and the buffer manager uses these to store the buffer for later recall. Subsequent records are handled as per step 2. 4. When the last record has been buffered, XML::Filter::Sort calls the buffer manager's "to_sax()" method. The buffer manager retrieves each of the buffers in sorted order and calls the buffer's "to_sax()" method. Each buffer attempts to match the sort key paths as SAX events are received. Once a value has been found for a given key, that same path match is not attempted against subsequent events. For efficiency, the code to match each key is compiled into a closure. For even more efficiency, this compilation is done once when the XML::Filter::Sort object is created. The "compile_matches()" method in the buffer manager class calls the "compile_matches()" method in the buffer class to achieve this. DATA STRUCTURES
In the current implementation, the XML::Filter::Sort::BufferMgr class simply uses a hash to store the buffer objects. If only one sort key was defined, only a single hash is required. The values in the hash are arrayrefs containing the list of buffers for records with identical keys. If two or more sort keys are defined, the hash values will be XML::Filter::Sort::BufferMgr objects which in turn will contain the buffers. The following illustration may clarify the relationship (BM=buffer manager, B=buffer): BM +----------------+---------------+ | | BM BM +-----+--------+ +-----+----------+ | | | | BM BM BM BM +-----+----+ +----+------+ +----+----+ +------+------+ | | | | | | | | | | | | [B,B,B] [B] [B,B] [B] [B,B] [B,B,B] [B] [B,B] [B] [B,B] [B,B,B] [B,B] This layered storage structure is transparent to the XML::Filter::Sort object which instantiates and interacts with only one buffer manager (the one at the top of the tree). COPYRIGHT
Copyright 2002 Grant McLean <grantm@cpan.org> This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.12.4 2002-06-14 XML::Filter::Sort::BufferMgr(3pm)
All times are GMT -4. The time now is 02:14 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy