Shell script failing to read large Xml record-urgent critical help


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
# 1  
Shell script failing to read large xml record with many records-Critical urgent help

Hi All,

I have shell script running on AIX 5.3 box. It has 7 to 8 "sed" commands piped(|) together. It has a an Xml file as its input which has many records internally. There are certain record which which have more than hundered tags.The script is taking a huge amount of time more than 1.5 hrs to process the whole xml fileSmilie.Can any one tell me what is the issue with a solution. I am assuming that lot of swapping is occuring from RAM to Disk and thus it is taking so long a time???

CAn any one suggest if i can allocate memory for this script process at the time of execution???

Thanks in advance!!

Pls respond early!! this is critical!!Smilie
# 2  
Shell script failing to read large Xml record-urgent critical help

Hi All,

I have shell script running on AIX 5.3 box. It has 7 to 8 "sed" commands piped(|) together. It has a an Xml file as its input which has many records internally. There are certain record which which have more than hundered tags.The script is taking a huge amount of time more than 1.5 hrs to process the whole xml fileSmilie.Can any one tell me what is the issue with a solution. I am assuming that lot of swapping is occuring from RAM to Disk and thus it is taking so long a time???

CAn any one suggest if i can allocate memory for this script process at the time of execution???

Thanks in advance!!

Pls respond early!! this is critical!!Smilie
# 3  
To give you sound advice it would be necessary to know what the script is, what your input data is and what your desired output is. Post it here (not *all* the data, just a significant sample) and we will think about it.

Lacking any info the best we can do is tell you some generalized hints which may or may not help in your specific case. Here is one: Probably the several sed-calls could be combined to one single script if they are piped one into another. This might speed things considerably.

Another one: maybe you are doing something context-oriented. sed is poor in that and maybe some of its shortcomings are covered by shell constructs. If this is the case you might be better of writing the whole in awk, which will be slower than sed in what sed can do well, but faster than sed and some shell constructs connected in a pipeline.

(For instance: if you try to cut out some part of every line sed is probably faster than then often-seen "awk '{print $5}'", but if the part you are cutting is a number and you want all these numbers totalled at the end than awk is quite better than cutting with sed and adding in a shell loop.)

bakunin
# 4  
Hi

i ahve all the sed's piped in the same script and my data consist of multiple Xml records in a xml file and the all the seds in total are taking long time.

I have taken 1 sed command singly and it took 25 mins... and i have 6 more piped along

i don't have much of an option to modify the script... other than something like ....

can we allocate memory (RAM) space [b'coz i have a very very huge RAM] so that swaping of the records may be reduced...

can u suggest something!!!plz plz
# 5  
As bakunin says, post your script so people can see what might be redundant or can be written more efficient. And again no no no, there is no allocating memory for shell scripts on AIX, not even on other derivates I think.
You can check if your ulimits might be set too low. Else tune your machine so it won't swap with vmo. But this is another issue.

And please do not spam "plz" - it sounds like a 14 year old in World of Warcraft. Sorry, but couldn't resist.
# 6  
Quote:
It has 7 to 8 "sed" commands piped(|) together.
One suggestion what I could say is ( not complete though )

if you are using sed commands like

Code:
sed 's/abc/def/g' filename | sed 's/rty/uty/g'

it can be combined as

Code:
sed 's/abc/def/g;s/rty/uty/g' filename

Second one would be much faster, as there is only one sed command ( process ) and iterations going on ( no Kernel Data Structure even )

Quote:
Pls respond early!! this is critical!!
This is a relative term and not encouraged here.
# 7  
Every time you execute a pipe you create a child process. This is expensive and consumes huge amounts of resource.

You can get past this by doing things like creating co-processes, or setting up cooperating processes. Again we need to see the "seven" pipes to sed. Which is very likely the bottleneck - but that is a guess.
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

UNIX shell script required to read two columns from xml

Hello All, I am new to unix scripting. I need to read FROMINSTANCE, FROMFIELD from below XML code and need to write a script for insert into SQ_EPIC values( "DID", "PROJECT_NAME") Select DID, PROJECT_NAME from EPIC<CONNECTOR TOINSTANCETYPE="Source Qualifier" TOINSTANCE="SQ_EPIC" TOFIELD="DID"... (7 Replies)
Discussion started by: sekhar.lsb
7 Replies

2. Shell Programming and Scripting

Read xml tags and then remove the tag using shell script

<Start> <Header> This is header section </Header> <Body> <Body_start> This is body section <a> <b> <c> <st>111</st> </c> <d> <st>blank</st> </d> </b> </a> </Body_start> <Body_section> This is body section (3 Replies)
Discussion started by: RJG
3 Replies

3. Red Hat

How to read an xml file through shell script?

Hey , can we read an xml file and make changes in it through shell script. Thanks (4 Replies)
Discussion started by: ramsavi
4 Replies

4. Shell Programming and Scripting

Shell Script to read XML file

Hi unix Gurus, I am really new to Unix Scripting. Please help me to create a shell script which reads the xml file and from that i need to fetch a particular information. For example <SOURCE BUSINESSNAME ="" DATABASETYPE ="Teradata" DBDNAME ="DWPROD3" DESCRIPTION ="" NAME... (5 Replies)
Discussion started by: SmilePlease
5 Replies

5. UNIX for Advanced & Expert Users

Shell Script to read XML tags and the data within that tag

Hi unix Gurus, I am really new to Unix Scripting. Please help me to create a shell script which reads the xml file and from that i need to fetch a particular information. For example <SOURCE BUSINESSNAME ="" DATABASETYPE ="Teradata" DBDNAME ="DWPROD3" DESCRIPTION ="" NAME... (2 Replies)
Discussion started by: SmilePlease
2 Replies

6. Shell Programming and Scripting

Urgent!! : How to read very long line using shell script

I have to fix files with only one huge line. Basically this line contains huge number of records may be in thousands of fix length. Suppose if we consider the length of one record as 100 bytes and if we have 50 such record. Then the file will have only one line with 100*50 bytes. I want to... (3 Replies)
Discussion started by: hrvispute
3 Replies

7. UNIX for Advanced & Expert Users

How to read an Xml record contained in a file--urgent

Hi I have an xml file which has multiple xml records.. I don't know how to read those records and pipe them to another shell command the file is like <abc>z<def>y<ghi>x........</ghi></def></abc> (1st record) <jkl>z<mno>y<pqr>x........</pqr></mno></jkl> (2nd record) Each record end... (4 Replies)
Discussion started by: aixjadoo
4 Replies

8. UNIX for Advanced & Expert Users

Allocate memory for a shell script in Aix at runtime-urgent critical

How to allocate memory for a shell script on aix box at the time of execution i.e at runtime Are there any commands for AIX in specific Thanks in Advance (1 Reply)
Discussion started by: aixjadoo
1 Replies

9. UNIX for Dummies Questions & Answers

allocate memory for a shell script at runtime--urgent critical help!!!

How to allocate memory for a shell script on aix box at the time of execution i.e at runtime Are there any commands for AIX in specific Thanks in Advance (3 Replies)
Discussion started by: aixjadoo
3 Replies

10. AIX

allocate memory for shell script at runtime during execution--urgent critical help!!

How to allocate memory for a shell script on aix box at the time of execution i.e at runtime Are there any commands for AIX in specific Thanks in Advance (1 Reply)
Discussion started by: aixjadoo
1 Replies

Featured Tech Videos