Moving XML tag/contents after specific XML tag within same file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Moving XML tag/contents after specific XML tag within same file
# 15  
Old 08-16-2018
Rant

Well. I'm always interested in competing to awk with other languages. I obviously can not compete in brevity(which is very impressive, when I see the solutions presented in this forum - but they may twist my brains sometimes which seems a horror to me, when coming back to a solution: WTF did I think, when I wrote that pile of crazy code?) so far, but I try to do in maintainability, efficiency(IO-request and memory economy) and runtime speed:

I don't know if you even are able to use ruby, but here's my suggestion in ruby(Just for the fun of learning).


/Rant


Code:
#!/usr/bin/env ruby

$handle = File.open(ARGV[0],"r")
$current_line = ""

$ah     ="^\s*<accountHolderName>.*<\/accountHolderName>\s*\n"
$aah    ="^\s*<additionalaccountHolders>.*<\/additionalaccountHolders>\s*\n"

def chunks()     
        Enumerator.new do |chunk|
             loop do
                     new = get_chunk()
                     if !new then break end
                     chunk << new
              end
        end
end

def get_chunk()
        while !$handle.eof do
                current_chunk = ( current_chunk ? current_chunk : "") + $current_line
                $current_line = $handle.readline
                if $current_line.match(/holder>/) then
                        return current_chunk
                end
        end
        return current_chunk
end

def reorder(chunk)
        ah_current=chunk.match(/(#{$ah})/im)
        aah_current=chunk.match(/#{$aah}/im)
        chunk.gsub!(/#{$aah}/im,"")
        chunk.gsub!(/#{$ah}/,"#{ah_current}#{aah_current}")
        return chunk
end

chunks.each {|c| puts reorder(c)}

Use it like this:

Code:
reorder.rb data.xml

------ Post updated at 01:56 PM ------

Or here with OOP:

Code:
#!/usr/bin/env ruby

class ChunkCollection
        def initialize(file)
                @handle = File.open(file,"r")
                @current_line=@data=""
        end
        def chunks()
                Enumerator.new do |c|
                     loop do
                             if data = get_chunk() then
                                     c << Chunk.new(data) 
                             else 
                                     break
                             end
                      end
                end
        end
        def get_chunk()
                while !@handle.eof do
                        current_chunk = ( current_chunk ? current_chunk : "") + @current_line
                        @current_line = @handle.readline
                        return current_chunk if @current_line.match(/holder>/) 
                end
        end
        def reorder() 
                chunks.each {|c| @data+=c.reorder()}
                return self
        end
        def show() puts @data end
end     

class Chunk
        def initialize(data) 
                @ah     ="^\s*<accountHolderName>.*<\/accountHolderName>\s*\n"
                @aah    ="^\s*<additionalaccountHolders>.*<\/additionalaccountHolders>\s*\n"
                @chunk = data
        end
        def reorder()
                ah_current=@chunk.match(/(#{@ah})/im)
                aah_current=@chunk.match(/#{@aah}/im)
                return @chunk.gsub(/#{@aah}/im,"").
                        gsub(/#{@ah}/,"#{ah_current}#{aah_current}")
        end
end

ChunkCollection.new(ARGV[0]).reorder.show

------ Post updated at 03:26 PM ------

I would suggest this little change to vgersh solution:


Quote:
Originally Posted by vgersh99
------ Post updated at 06:23 PM ------
Code:
awk '
  /<accountHolderName/ {accH=FNR}
  FNR==NR {
     if (/<AdditionalAccountHolders/) {s[accH]=(s[accH])?s[accH] ORS $0:$0;f++;next}
     if (f) s[accH]= s[accH] ORS $0
     if (/<[/]AdditionalAccountHolders/) f--
     next
  }
  /<accountHolderName/ && s[accH] {print $0 ORS s[accH];next}
  /<AdditionalAccountHolders/,/<[/]AdditionalAccountHolders/ {next}
  1' myXMLfile myXMLfile


Edit

My change is not needed. Even the umodified version prior to the input data specification change (without f++ but f=1) works.


And well: That awk solution is not really that complicated.... Smilie

Last edited by stomp; 08-16-2018 at 10:40 AM..
# 16  
Old 08-16-2018
Awesome!!

Thanks to both Stomp and vgersh99 for spending your time and helping me out. This latest awk code works magically great and the results obtained are as expected:

Code:
awk '
  /<accountHolderName/ {accH=FNR}
  FNR==NR {
     if (/<AdditionalAccountHolders/) {s[accH]=(s[accH])?s[accH] ORS $0:$0;f++;next}
     if (f) s[accH]= s[accH] ORS $0
     if (/<[/]AdditionalAccountHolders/) f--
     next
  }
  /<accountHolderName/ && s[accH] {print $0 ORS s[accH];next}
  /<AdditionalAccountHolders/,/<[/]AdditionalAccountHolders/ {next}
  1' myXMLfile myXMLfile

# 17  
Old 08-16-2018
As a systems guy, i would always choose awk, if possible.
Not perhaps for parsing xml...

Examine the strace -c <code> reports :
For awk :
Code:
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 27.26    0.000558          13        42           write
 18.61    0.000381          21        18           read
 11.87    0.000243          16        15           openat
  5.72    0.000117          39         3           munmap
  5.28    0.000108           5        20           fstat
  4.49    0.000092          46         2           lseek
  4.35    0.000089           3        26           mmap
  4.25    0.000087          15         6           brk
  3.71    0.000076           5        15           close
  3.13    0.000064          64         1           stat
  2.54    0.000052          10         5           rt_sigaction
  2.00    0.000041           2        18           mprotect
  1.91    0.000039           7         6           fcntl
  1.27    0.000026           9         3         2 ioctl
  0.73    0.000015           8         2           getgroups
  0.44    0.000009           9         1           sigaltstack
  0.39    0.000008           8         1           getpgrp
  0.34    0.000007           7         1           getpid
  0.34    0.000007           7         1           getuid
  0.34    0.000007           7         1           getgid
  0.34    0.000007           7         1           geteuid
  0.34    0.000007           7         1           getegid
  0.34    0.000007           7         1           getppid
  0.00    0.000000           0        10        10 access
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         1           arch_prctl
------ ----------- ----------- --------- --------- ----------------
100.00    0.002047                   202        12 total

And for ruby :
Code:
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 37.68    0.003260           7       459       320 openat
 18.30    0.001583           4       361         7 lstat
 12.39    0.001072           6       186           read
  6.60    0.000571           4       141           close
  4.91    0.000425           3       141           fstat
  2.84    0.000246           3        76         1 stat
  2.53    0.000219           3        74        71 ioctl
  2.35    0.000203           3        61           fcntl
  2.24    0.000194           4        47           getegid
  2.09    0.000181           5        38           brk
  1.83    0.000158           3        46           getgid
  1.68    0.000145           3        46           getuid
  1.41    0.000122           3        47           geteuid
  1.33    0.000115          19         6           getdents
  0.89    0.000077           2        38           mmap
  0.47    0.000041           2        27           mprotect
  0.43    0.000037           2        24           lseek
  0.02    0.000002           2         1           sysinfo
  0.00    0.000000           0         5           write
  0.00    0.000000           0         4           munmap
  0.00    0.000000           0        18           rt_sigaction
  0.00    0.000000           0         3           rt_sigprocmask
  0.00    0.000000           0        12        12 access
  0.00    0.000000           0         3           getpid
  0.00    0.000000           0         1           clone
  0.00    0.000000           0         3         1 execve
  0.00    0.000000           0         1           getcwd
  0.00    0.000000           0         1           sigaltstack
  0.00    0.000000           0         2           arch_prctl
  0.00    0.000000           0         1           futex
  0.00    0.000000           0         1           sched_getaffinity
  0.00    0.000000           0         1           set_tid_address
  0.00    0.000000           0         1           clock_gettime
  0.00    0.000000           0         1           set_robust_list
  0.00    0.000000           0         2           pipe2
  0.00    0.000000           0         3           prlimit64
  0.00    0.000000           0         2           getrandom
------ ----------- ----------- --------- --------- ----------------
100.00    0.008651                  1884       412 total

The difference will not be noticed on a system when parsing one file.
But in a situation where you need to parse tens of thousands ...

Possibly the ruby code can be written to do it better, but doubtful it will ever surpass awk code in performance.
This is to say if two ideal coders write a program in ruby and awk to do one thing best it can and start forking it Smilie

So, to conclude, in my opinion higher level languages are to be used in situations where your program needs many libraries to ease up the job - connect to multiple API endpoints, databases, versioning systems, complex math and such.

You could do all that in awk, but tremendous effort will be required and will beat the propose of short programs which do one thing quick and efficient.

This is just my rant Smilie
Regards
Peasant.
These 3 Users Gave Thanks to Peasant For This Post:
# 18  
Old 08-16-2018
noted that the awk solution is in this case far ahead even it is used in that inefficient way that it is reading the input file twice...


Thanks for "strace -c". Never used that before. Good bloat indicator.

------ Post updated at 05:51 PM ------

Well. I thought I did my ruby script fairly good, but it's an absolute desaster. I generated an xml data file of just 10 MB. This is the result.

AWK Resources
Code:
0.50user 0.00system 0:00.56elapsed 89%CPU (0avgtext+0avgdata 7344maxresident)k
 2592inputs+0outputs (10major+1130minor)pagefaults 0swaps

7860 system calls

Ruby procedural
Code:
5.29user 0.06system 0:05.38elapsed 99%CPU (0avgtext+0avgdata 13604maxresident)k
0inputs+0outputs (0major+2523minor)pagefaults 0swaps

4902 system calls

Ruby OOP
Code:
41.35user 36.62system 1:18.19elapsed 99%CPU (0avgtext+0avgdata 254192maxresident)k
352inputs+0outputs (3major+22449045minor)pagefaults 0swaps

 37602 system calls

I assume the string-concatenation is really bad here.


Ruby OOP(without storing the result in memory)
Code:
4.93user 0.31system 0:05.57elapsed 94%CPU (0avgtext+0avgdata 13264maxresident)k
0inputs+0outputs (0major+2611minor)pagefaults 0swaps

  4901 system calls

So if one wants speed and low memory footprint, one can tune a lot with [high-level-programming-language] or just take awk Smilie

In terms of system calls ruby won here(probably because of the double reading of the file with awk) but it's 10 times slower. I think has some fat base whereas awk is very lean, so as more complex the task is, the less relevant is the basic bloat.

Last edited by stomp; 08-16-2018 at 02:05 PM.. Reason: Fixed false results
# 19  
Old 08-16-2018
When measuring, couple of things to consider.

Todays OS and filesystems are smart, they cache, prefetch and similar math magic being done falling into probability and combinatorics division.
So far and deep in HW that they give you other users data when asked nicely Smilie
Filesystems will cache the first 10 MB read, so second read will be amazingly fast(er).

Be sure to take above into consideration during testing.

This was not done to compare ruby or awk per se, just to point out not to limit yourself to certain path, but use the right tool for the task.

As for the strace options, i've read the manual a bit before, to find an option, since i was sure GNU stuff has that nicely formatted without effort Smilie

Regards
Peasant.
# 20  
Old 08-16-2018
Quote:
Filesystems will cache the first 10 MB read, so second read will be amazingly fast(er).
Of course. I assume cache is voiding any significant normal read times here. I can create additional processing overhead by reading in too small portions or improve performance by reading larger chunks. This is good, because so now the times here are processing times only.

Quote:
This was not done to compare ruby or awk per se, just to point out not to limit yourself to certain path, but use the right tool for the task.
My curiosity here is NOT "the right tool for the right job" at the moment. My point is: Is [some high-level-programming-language] too bloated and not able to compete in this single task with awk in terms of speed? If not, how much it is behind?


I already tested the same algorithm which is used for awk here in ruby. It's roughly 3 times faster(still 2-3 times slower than awk), but far less elegant than the awk code. That's a first interesting insight. Along with the other realization that line based processing seems to be a lot faster than my chunk-based processing. I've got an idea too, what of my codeparts are a worse and it is good to see actually how much the difference for those "little" things is.

Last edited by stomp; 08-16-2018 at 03:49 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Grepping multiple XML tag results from XML file.

I want to write a one line script that outputs the result of multiple xml tags from a XML file. For example I have a XML file which has below XML tags in the file: <EMAIL>***</EMAIL> <CUSTOMER_ID>****</CUSTOMER_ID> <BRANDID>***</BRANDID> Now I want to grep the values of all these specified... (1 Reply)
Discussion started by: shubh752
1 Replies

2. Shell Programming and Scripting

To search for a particular tag in xml and collate all similar tag values and display them count

I want to basically do the below thing. Suppose there is a tag called object1. I want to display an output for all similar tag values under heading of Object 1 and the count of the xmls. Please help File: <xml><object1>house</object1><object2>child</object2>... (9 Replies)
Discussion started by: srkmish
9 Replies

3. Shell Programming and Scripting

XML Parse between to tag with upper tag

Hi Guys Here is my Input : <?xml version="1.0" encoding="UTF-8"?> <xn:MeContext id="01736"> <xn:VsDataContainer id="01736"> <xn:attributes> <xn:vsDataType>vsDataMeContext</xn:vsDataType> ... (12 Replies)
Discussion started by: pareshkp
12 Replies

4. Shell Programming and Scripting

How to add the multiple lines of xml tags before a particular xml tag in a file

Hi All, I'm stuck with adding multiple lines(irrespective of line number) to a file before a particular xml tag. Please help me. <A>testing_Location</A> <value>LA</value> <zone>US</zone> <B>Region</B> <value>Russia</value> <zone>Washington</zone> <C>Country</C>... (0 Replies)
Discussion started by: mjavalkar
0 Replies

5. Shell Programming and Scripting

How to retrieve the value from XML tag whose end tag is in next line

Hi All, Find the following code: <Universal>D38x82j1JJ </Universal> I want to retrieve the value of <Universal> tag as below: Please help me. (3 Replies)
Discussion started by: mjavalkar
3 Replies

6. Shell Programming and Scripting

how to retrieve specific parameters using a xml tag

Hi, I have the following code in my xml file: <aaaRule loginIdPattern=".*" orgIdPattern=".*" deny="false" /> <aaaRuleGroup name="dpaas"> <aaaRule loginIdPattern=".*" orgIdPattern=".*" deny="false" /> I want to retrieve orgIdPattern and loginIdPattern parameter value based on... (2 Replies)
Discussion started by: mjavalkar
2 Replies

7. Shell Programming and Scripting

XML tag replacement from different XML file

We have 2 XML file 1. ORIGINAL.xml file and 2. ATTRIBUTE.xml files, In the ORIGINAL.xml we need some modification as <resourceCode>431048</resourceCode>under <item type="Manufactured"> tag - we need to grab the 431048 value from tag and pass it to database table in unix shell script to find the... (0 Replies)
Discussion started by: balrajg
0 Replies

8. Shell Programming and Scripting

Extracting XML Tag Contents

Hi Jean I require your help in writing a shell script. Iam zero in Unix programming. I have a large file about 400 MB of data, which contains about 50000 XML messages seperated by a Tab, I think. I need to extract only 4 values from each XML message and write it onto a new file. Please help me... (2 Replies)
Discussion started by: pk_eee
2 Replies

9. Shell Programming and Scripting

how to get specific xml tag ?

sorry to trouble u guys again... i have this document here called record.txt and it contains this: 2005-12-05 10:53:17,551 INFO - message received... 2005-12-05 10:53:17,557 INFO - The XML message **************<berth_allocation xmln... (13 Replies)
Discussion started by: forevercalz
13 Replies
Login or Register to Ask a Question