Sponsored Content
Full Discussion: sed performance
Top Forums UNIX for Advanced & Expert Users sed performance Post 302174498 by f3k on Tuesday 11th of March 2008 09:11:37 AM
Old 03-11-2008
sed performance

hello experts,

i am trying to replace a line in a 100+mb text file. the structure is similar to the passwd file, id:value1:value2 and so on. using the sed command

Code:
sed -i 's/\(123\):\([^:]\{1,\}\):/\1:bar:/' data.txt

works nicely, the line "123:foo:" is replaced by "123:bar:". however, it takes about 3 seconds.

using the grep command with environment variable LC_ALL set to "C" brings me the result instantly:

Code:
export LC_ALL=C
grep -Eh -- '(123):([^:]{1,}):' data.txt

now, the occurrences i'm looking for are all near the end of the file, so performance loss due to i/o could be avoided, since not the whole file should be rewritten. does anyone have an idea how sed could be accelerated? several ideas pop into my mind:
  • make sed somehow seek to the byte offset grep can deliver rapidly
  • only pipe the tail starting at the occurrence to sed and somehow write the tail back to the file
  • find out why sed is so slow, maybe it is matching in unicode (without LC_ALL=C, grep is slow too)

any hints or leads would be greatly appreciated Smilie

cheers,

-f3k.
 

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Performance

Hello, i have changed a slow server with Solaris 7 to a bigger one with Solaris 8 (Sun Ultra 2). Now i have a real bad performance problem (only CPU). Solaris 7 ran with standard FTP and Samba 2.0.7. The new machine is running ProFTP and Samba 2.0.9. There are a lot of NFS Shares and... (5 Replies)
Discussion started by: olso
5 Replies

2. UNIX for Advanced & Expert Users

performance

Hi, 1-in vmstat commande line, in reply, which column is the more important to look and verify if server is very slow ? 2-how can I see how many sessions are opened with the same login ? Many thanks before. (2 Replies)
Discussion started by: big123456
2 Replies

3. UNIX for Advanced & Expert Users

performance

Hi, I have this on a AIX UNIX machine : ps aux| head -20 USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND root 516 23.7 0.0 12 15808 - A 19:38:15 903:13 wait root 774 23.7 0.0 12 15808 - A 19:38:15 902:13 wait root 1290 23.6 0.0 ... (2 Replies)
Discussion started by: big123456
2 Replies

4. UNIX for Advanced & Expert Users

I/O performance

i want to determine I/O performance of an executable, but iostat dont give correct results because the disk that i am writing to and reading from, are not physical disk of the host machine, instead of these local disks we are using a network storage. is there any standard way in unix to get... (2 Replies)
Discussion started by: gfhgfnhhn
2 Replies

5. News, Links, Events and Announcements

Announcing collectl - new performance linux performance monitor

About 4 years ago I wrote this tool inspired by Rob Urban's collect tool for DEC's Tru64 Unix. What makes this tool as different as collect was in its day is its ability to run at a low overhead and collect tons of stuff. I've expanded the general concept and even include data not available in... (0 Replies)
Discussion started by: MarkSeger
0 Replies

6. Solaris

best way and best performance

Hi all, I have two storadge 3510 Fc .. with 12 disks 146Gb ..total 1752Gb each storadge. I need to use about 1.4 Tb of it. and want RAID1 .. I need 13 mount points .. So question: for best performance and redundjancy how I must do it. create 13 logical drives on each stordge with same size... (1 Reply)
Discussion started by: samar
1 Replies

7. Shell Programming and Scripting

Increase sed performance

I'm using sed to do find and replace. But since the file is huge and i have more than 1000 files to be searched, the script is taking a lot of time. Can somebody help me with a better sed command. Below is the details. Input: 1 1 2 3 3 4 5 5 Here I know the file is sorted. ... (4 Replies)
Discussion started by: gpaulose
4 Replies

8. Shell Programming and Scripting

sed / grep / for statement performance - please help

I'm searching the most effective way of doing the following task, so if someone can either provide a working solution with sed or one totally different but more effective then what I've got so far then please go ahead! The debugme directory has 3 subdirectorys and each of them has one .txt file... (7 Replies)
Discussion started by: TehOne
7 Replies

9. UNIX for Dummies Questions & Answers

Performance analysis sed vs awk

Hi Guys, I've wondered for some time the performance analysis between using sed and awk. say i want to print lines from a very large file. For ex say a file with 100,000 records. i want to print the lines 25,000 to 26,000 i can do so by the following commands: sed -n '25000,26000 p'... (11 Replies)
Discussion started by: Irishboy24
11 Replies
ZGREP(1)						      General Commands Manual							  ZGREP(1)

NAME
zgrep - search possibly compressed files for a regular expression SYNOPSIS
zgrep [ grep_options ] [ -e ] pattern filename... DESCRIPTION
Zgrep invokes grep on compressed or gzipped files. These grep options will cause zgrep to terminate with an error code: (-[drRzZ]|--di*|--exc*|--inc*|--rec*|--nu*). All other options specified are passed directly to grep. If no file is specified, then the standard input is decompressed if necessary and fed to grep. Otherwise the given files are uncompressed if necessary and fed to grep. If the GREP environment variable is set, zgrep uses it as the grep program to be invoked. EXIT CODE
2 - An option that is not supported was specified. AUTHOR
Charles Levert (charles@comm.polymtl.ca) SEE ALSO
grep(1), gzexe(1), gzip(1), zdiff(1), zforce(1), zmore(1), znew(1) ZGREP(1)
All times are GMT -4. The time now is 04:10 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy