sed Very Slow

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sed Very Slow
# 1  
Old 10-11-2013
sed Very Slow


We are using sed to clean up a file of a pattern and its talking a lot of time on XML output file

The command that we are using is

sed -e "s/tns1://g" $OUTPUTFILENM > $TEMPFILE

Where $OUTPUTFILENM is the file to be cleaned and $TEMPFILE is the cleaned output

Can you please help me to optimise this command or suggest some other command that can clean up say the string tns1: and replace it with space across all xml file.

I will really appreciate any suggestion. For a 45k record file its taking more than an hour in each record we have around 10 instances of string tns:

Also the string can appear any where its not fixed width location since we have XML file that we are cleanning


Last edited by Scott; 10-11-2013 at 06:35 PM.. Reason: Icode tags -> Code tags
# 2  
Old 10-11-2013
Something is very wrong here. sed can process 10+ GB's/hour on a modern machine.
Please show the output of:

ls -l [one of your xml files]
wc -l [the same xml file]

# 3  
Old 10-11-2013
Here are the outputs of the above commands

-rw-rw-rw-    1 inf  inf    103876292 Oct 11 18:16
wc -l returns 0 as its a single root tag XML file

Last edited by Scrutinizer; 10-12-2013 at 04:51 AM.. Reason: changed icode tags to regular code tags
# 4  
Old 10-12-2013
That looks like one 103 MB long line.
sed is not optimized for this.
Try another sed version,
or perl
perl -pe "s/tns1://g" $OUTPUTFILENM > $TEMPFILE

This User Gave Thanks to MadeInGermany For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

7 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Why is SED so slow?

I have many files which contain about two million lines. Now I want to use sed to delete the 9th line and add a new line behind the 8th line. I use the command as follows: for((i=1;i<100;i++)); do echo $i; sed -i '9d' $i.dat; sed -i '8a this is a new line' $i.dat; done But it is... (3 Replies)
Discussion started by: wxuyec
3 Replies

2. UNIX for Advanced & Expert Users

sed working slow on big files

HI Experts , I'm using the following code to remove spaces appearing at the end of the file. sed "s/*$//g" <filename> > <new_filename> mv <new_filename> <filename> this is working fine for volumes upto 20-25 GB. for the bigger files it is taking more time that it is required... (5 Replies)
Discussion started by: sumoka
5 Replies

3. Shell Programming and Scripting

cut, sed, awk too slow to retrieve line - other options?

Hi, I have a script that, basically, has two input files of this type: file1 key1=value1_1_1 key2=value1_2_1 key4=value1_4_1 ... file2 key2=value2_2_1 key2=value2_2_2 key3=value2_3_1 key4=value2_4_1 ... My files are 10k lines big each (approx). The keys are strings that don't... (7 Replies)
Discussion started by: fzd
7 Replies

4. Shell Programming and Scripting

Cut too slow

Hi I am using a cut command in the script which is slowing down the performance of the script .can anyone suggest the other ways of doing the same cut command Record_Type=`echo "$line" | cut -c19-20` *******this is slowing down********* i have 4 more cut commands 2 in one loop and 2 in inner... (3 Replies)
Discussion started by: pukars4u
3 Replies

5. Solaris

Slow Login

Hi All, I have problem when i write my user name to login to my server late (about 10 min) to give me field of password if u know how i can solve it? Thanks (4 Replies)
Discussion started by: mass1123
4 Replies

6. UNIX for Dummies Questions & Answers

Slow System

Hi, I have an SCO-Unix server running. There are some processes (unknown to me) which consume a lot of the system resources. This slows down the server dramatically. Is there a command or program that monitors what processes are using the cpu, disk, etc.. and tell me how excessive and how... (3 Replies)
Discussion started by: Hansaplast
3 Replies

7. Post Here to Contact Site Administrators and Moderators


The site has gone slow for quite some time... Can you do somethin abt it (2 Replies)
Discussion started by: DPAI
2 Replies
Login or Register to Ask a Question

Featured Tech Videos