sed Very Slow


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sed Very Slow
# 1  
Old 10-11-2013
sed Very Slow

Hi

We are using sed to clean up a file of a pattern and its talking a lot of time on XML output file

The command that we are using is

Code:
sed -e "s/tns1://g" $OUTPUTFILENM > $TEMPFILE

Where $OUTPUTFILENM is the file to be cleaned and $TEMPFILE is the cleaned output

Can you please help me to optimise this command or suggest some other command that can clean up say the string tns1: and replace it with space across all xml file.

I will really appreciate any suggestion. For a 45k record file its taking more than an hour in each record we have around 10 instances of string tns:

Also the string can appear any where its not fixed width location since we have XML file that we are cleanning

Thanks

Last edited by Scott; 10-11-2013 at 06:35 PM.. Reason: Icode tags -> Code tags
# 2  
Old 10-11-2013
Something is very wrong here. sed can process 10+ GB's/hour on a modern machine.
Please show the output of:

Code:
ls -l [one of your xml files]
wc -l [the same xml file]

# 3  
Old 10-11-2013
Here are the outputs of the above commands

Code:
-rw-rw-rw-    1 inf  inf    103876292 Oct 11 18:16
wc -l returns 0 as its a single root tag XML file


Last edited by Scrutinizer; 10-12-2013 at 04:51 AM.. Reason: changed icode tags to regular code tags
# 4  
Old 10-12-2013
That looks like one 103 MB long line.
sed is not optimized for this.
Try another sed version,
or perl
Code:
perl -pe "s/tns1://g" $OUTPUTFILENM > $TEMPFILE

This User Gave Thanks to MadeInGermany For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

7 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Why is SED so slow?

I have many files which contain about two million lines. Now I want to use sed to delete the 9th line and add a new line behind the 8th line. I use the command as follows: for((i=1;i<100;i++)); do echo $i; sed -i '9d' $i.dat; sed -i '8a this is a new line' $i.dat; done But it is... (3 Replies)
Discussion started by: wxuyec
3 Replies

2. UNIX for Advanced & Expert Users

sed working slow on big files

HI Experts , I'm using the following code to remove spaces appearing at the end of the file. sed "s/*$//g" <filename> > <new_filename> mv <new_filename> <filename> this is working fine for volumes upto 20-25 GB. for the bigger files it is taking more time that it is required... (5 Replies)
Discussion started by: sumoka
5 Replies

3. Shell Programming and Scripting

cut, sed, awk too slow to retrieve line - other options?

Hi, I have a script that, basically, has two input files of this type: file1 key1=value1_1_1 key2=value1_2_1 key4=value1_4_1 ... file2 key2=value2_2_1 key2=value2_2_2 key3=value2_3_1 key4=value2_4_1 ... My files are 10k lines big each (approx). The keys are strings that don't... (7 Replies)
Discussion started by: fzd
7 Replies

4. Shell Programming and Scripting

Cut too slow

Hi I am using a cut command in the script which is slowing down the performance of the script .can anyone suggest the other ways of doing the same cut command Record_Type=`echo "$line" | cut -c19-20` *******this is slowing down********* i have 4 more cut commands 2 in one loop and 2 in inner... (3 Replies)
Discussion started by: pukars4u
3 Replies

5. Solaris

Slow Login

Hi All, I have problem when i write my user name to login to my server late (about 10 min) to give me field of password if u know how i can solve it? Thanks (4 Replies)
Discussion started by: mass1123
4 Replies

6. UNIX for Dummies Questions & Answers

Slow System

Hi, I have an SCO-Unix server running. There are some processes (unknown to me) which consume a lot of the system resources. This slows down the server dramatically. Is there a command or program that monitors what processes are using the cpu, disk, etc.. and tell me how excessive and how... (3 Replies)
Discussion started by: Hansaplast
3 Replies

7. Post Here to Contact Site Administrators and Moderators

Slow

The site has gone slow for quite some time... Can you do somethin abt it (2 Replies)
Discussion started by: DPAI
2 Replies
Login or Register to Ask a Question