Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Duplicates in an XML file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Duplicates in an XML file
# 1  
Old 08-31-2011
Duplicates in an XML file
Hi All,

I have an xml file that contains information like this

Code:
 
<ID>574922<COMMENT>TEXT
TEXT
TEXT</COMMENT></ID>
<ID>574922<COMMENT>TEXT
TEXT
TEXT</COMMENT></ID>
<ID>412659<COMMENT>TEXT
TEXT
TEXT
TEXT
TEXT</COMMENT></ID>
<ID>873520<COMMENT>TEXT</COMMENT></ID>
<ID>480622<COMMENT>TEXT</COMMENT></ID>
<ID>873520<COMMENT>TEXT
TEXT</COMMENT></ID>
<ID>480622<COMMENT>TEXT</COMMENT></ID>

I want to remove duplicate entries, the problem is I cannot sort, as due to comment stracture for some entries, they form a new line.

I tried the following code
(called as awk -f script file)
Code:
/^end:/ {   if (! (Record in Records)) {      
Records[Record];      
print RecordLabel ":";      
print Record;      
print $0;        
Record = "";   
}   
next;}$1 ~ /^.*:/ {   sub(/:.*/, "", $1);   
RecordLabel = $1;   
next;}{   Record = (Record ? Record "\n" : "") $0;}

provided by Aigles in another post, which for some reason does not work (modified to my input), it does not change anything. I tried to variate it unsuccsfully so far.

any ideas would be much appreciated

many thanks
Last edited by TasosARISFC; 09-08-2011 at 05:34 AM..
# 2  
Old 08-31-2011
The usual way to do this is to use a modified form of Muenchian grouping in a XSLT 1.0 stylesheet.
# 3  
Old 08-31-2011
Hi, sadly I have no XSLT processor nor can I install one as my machine is restricted

---------- Post updated at 03:57 PM ---------- Previous update was at 03:53 PM ----------

Is there a way to check whats between <ID></ID> and check if that exists somewhere else in the file. if it does...delete it? Also how can I bypass "/" in awk? for example I cannot do this:

Code:
/^</ID>/

I want to search for </ID>
# 4  
Old 08-31-2011
Escape it with \, like \/

Unfortunately processing XML isn't trivial. Without a proper recursive parser for it you end up building one yourself, brute-force, character by character, because the record-based language constructs of awk don't help you. That's why tools like XSLT processors exist..

working on something.

---------- Post updated at 10:55 AM ---------- Previous update was at 09:06 AM ----------

Here's a semi-ugly GNU awk solution. It works by breaking apart records on < and fields on >. Meaning, the first token is always a complete tag and the second, if any, is text -- <stuff param=1>text would get split into "stuff param=1", "text". I've tried to make it tolerate improperly nested tags, uppercase vs lowercase, etc, but can't possibly make it perfect.

Code:
$ cat xml.awk
#!/usr/bin/gawk -f

function inside(STR, OFF, N)
{
        for(N=(D-(1+OFF)); N>=0; N--)
        if(TAG[N] == tolower(STR))      return(1);

        return(0);
}

BEGIN {         RS="<"  ;       FS=">"  ;       LAST="C"        }

{
        # closing tag
        if($0 ~ /^[/]/)
        {
                TMP=D;
                D--;
                # Try to deal gracefully with improperly nested tags
                while(tolower(substr($1, 2, length($1)-1)) != TAG[D])
                {
                        if(D > 0)       D--;
                        else    {       D=TMP;  break;          }
                }

                if(!inside("ID", 0))    printf("<%s>\n", $1);
                LAST="C"
        }
        else if($1)     # open tag
        {
                if(!inside("ID", 0))
                {
                        if(LAST == "O") { printf("\n"); }

                        printf("<%s>", $1);
                        LAST="O";
                }

                if(! ($1 ~ /[/]$/))     # ignore self-closing tags
                {
                        split($1, a, "[ \r\n\t]");
                        TAG[D++]=tolower(a[1]);
                }

                if(!inside("ID", 1))
                if(NF > 1)      printf("%s", $2);
        }
}
$ ./xml.awk < data.xml
<ID>574922</ID>
<ID>574922</ID>
<ID>412659</ID>
<ID>873520</ID>
<ID>480622</ID>
<ID>873520</ID>
<ID>480622</ID>
$

Last edited by Corona688; 08-31-2011 at 02:33 PM.. Reason: [edit] improved version with fewer extra newlines
# 5  
Old 09-08-2011
Hi Corona, thank you for your effort. As in your example what this did was to create a list of <ID> 0000000</ID> removing all other tags, but still with duplicates.

I thought of sorting and use uniq to get the duplicate IDs from this list, then delete them from the original file (not the list). However this does not resolve the problem that some elements have comments that extent to new lines and those lines do not get removed

eg

Code:
 
<ID> 000000<COMMENT> TEXT
TEXT
TEXT
TEXT </COMMENT></ID>

will only delete the first line

So again I am looking for a way to look for xml tags <ID></ID> and if whats between them already exists in the file delete it.

---------- Post updated at 10:11 AM ---------- Previous update was at 09:29 AM ----------

I can see this is rather complicated... an other way I could do this is by providing the list of duplicate ID's, search the file for them and delete them. I already know the duplicate IDs from a sys out, so I can place them in a text file as a list.However I have two issues

First, how do I delete all but one?
and second how do I define where to start deleting and where to stop?
# 6  
Old 09-12-2011
You could replace all newlines with spaces, then insert newlines only where you want them:

Code:
$ tr '\n' ' ' < data  | sed 's#</ID>#</ID>\n#g'
<ID>574922<COMMENT>TEXT TEXT TEXT</COMMENT></ID>
 <ID>574922<COMMENT>TEXT TEXT TEXT</COMMENT></ID>
 <ID>412659<COMMENT>TEXT TEXT TEXT TEXT TEXT</COMMENT></ID>
 <ID>873520<COMMENT>TEXT</COMMENT></ID>
 <ID>480622<COMMENT>TEXT</COMMENT></ID>
 <ID>873520<COMMENT>TEXT TEXT</COMMENT></ID>
 <ID>480622<COMMENT>TEXT</COMMENT></ID>
$

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to pull multiple XML tags from the same XML file in Shell.?

I'm searching for the names of a TV show in the XML file I've attached at the end of this post. What I'm trying to do now is pull out/list the data from each of the <SeriesName> tags throughout the document. Currently, I'm only able to get data the first instance of that XML field using the... (9 Replies)
Discussion started by: hungryd
9 Replies

2. UNIX for Beginners Questions & Answers

Grepping multiple XML tag results from XML file.

I want to write a one line script that outputs the result of multiple xml tags from a XML file. For example I have a XML file which has below XML tags in the file: <EMAIL>***</EMAIL> <CUSTOMER_ID>****</CUSTOMER_ID> <BRANDID>***</BRANDID> Now I want to grep the values of all these specified... (1 Reply)
Discussion started by: shubh752
1 Replies

3. Shell Programming and Scripting

Splitting a single xml file into multiple xml files

Hi, I'm having a xml file with multiple xml header. so i want to split the file into multiple files. Sample.xml consists multiple headers so how can we split these multiple headers into multiple files in unix. eg : <?xml version="1.0" encoding="UTF-8"?> <ml:individual... (3 Replies)
Discussion started by: Narendra921631
3 Replies

4. Shell Programming and Scripting

Split xml file into multiple xml based on letterID

Hi All, We need to split a large xml into multiple valid xml with same header(2lines) and footer(last line) for N number of letterId. In the example below we have first 2 lines as header and last line as footer.(They need to be in each split xml file) Header: <?xml version="1.0"... (5 Replies)
Discussion started by: vx04
5 Replies

5. Shell Programming and Scripting

Comparing delta values of one xml file in other xml file

Hi All, I have two xml files. One is having below input <NameValuePair> <name>Daemon</name> <value>tcp:7474</value> </NameValuePair> <NameValuePair> <name>Network</name> <value></value> </NameValuePair> ... (2 Replies)
Discussion started by: sharsour
2 Replies

6. Shell Programming and Scripting

XML: parsing of the Google contacts XML file

I am trying to parse the XML Google contact file using tools like xmllint and I even dived into the XSL Style Sheets using xsltproc but I get nowhere. I can not supply any sample file as it contains private data but you can download your own contacts using this script: #!/bin/sh # imports... (9 Replies)
Discussion started by: ripat
9 Replies

7. Shell Programming and Scripting

Help required in Splitting a xml file into multiple and appending it in another .xml file

HI All, I have to split a xml file into multiple xml files and append it in another .xml file. for example below is a sample xml and using shell script i have to split it into three xml files and append all the three xmls in a .xml file. Can some one help plz. eg: <?xml version="1.0"?>... (4 Replies)
Discussion started by: ganesan kulasek
4 Replies

8. Shell Programming and Scripting

How to add the multiple lines of xml tags before a particular xml tag in a file

Hi All, I'm stuck with adding multiple lines(irrespective of line number) to a file before a particular xml tag. Please help me. <A>testing_Location</A> <value>LA</value> <zone>US</zone> <B>Region</B> <value>Russia</value> <zone>Washington</zone> <C>Country</C>... (0 Replies)
Discussion started by: mjavalkar
0 Replies

9. Shell Programming and Scripting

How to remove xml namespace from xml file using shell script?

I have an xml file: <AutoData xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <Table1> <Data1 10 </Data1> <Data2 20 </Data2> <Data3 40 </Data3> <Table1> </AutoData> and I have to remove the portion xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" only. I tried using sed... (10 Replies)
Discussion started by: Gary1978
10 Replies

10. Shell Programming and Scripting

How to parse a XML file using PERL and XML::DOm

I need to know the way. I have got parsing down some nodes. But I was unable to get the child node perfectly. If you have code please send it. It will be very useful for me. (0 Replies)
Discussion started by: girigopal
0 Replies
Login or Register to Ask a Question