Sponsored Content
Full Discussion: sed performance
Top Forums UNIX for Advanced & Expert Users sed performance Post 302174498 by f3k on Tuesday 11th of March 2008 09:11:37 AM
Old 03-11-2008
sed performance

hello experts,

i am trying to replace a line in a 100+mb text file. the structure is similar to the passwd file, id:value1:value2 and so on. using the sed command

Code:
sed -i 's/\(123\):\([^:]\{1,\}\):/\1:bar:/' data.txt

works nicely, the line "123:foo:" is replaced by "123:bar:". however, it takes about 3 seconds.

using the grep command with environment variable LC_ALL set to "C" brings me the result instantly:

Code:
export LC_ALL=C
grep -Eh -- '(123):([^:]{1,}):' data.txt

now, the occurrences i'm looking for are all near the end of the file, so performance loss due to i/o could be avoided, since not the whole file should be rewritten. does anyone have an idea how sed could be accelerated? several ideas pop into my mind:
  • make sed somehow seek to the byte offset grep can deliver rapidly
  • only pipe the tail starting at the occurrence to sed and somehow write the tail back to the file
  • find out why sed is so slow, maybe it is matching in unicode (without LC_ALL=C, grep is slow too)

any hints or leads would be greatly appreciated Smilie

cheers,

-f3k.
 

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Performance

Hello, i have changed a slow server with Solaris 7 to a bigger one with Solaris 8 (Sun Ultra 2). Now i have a real bad performance problem (only CPU). Solaris 7 ran with standard FTP and Samba 2.0.7. The new machine is running ProFTP and Samba 2.0.9. There are a lot of NFS Shares and... (5 Replies)
Discussion started by: olso
5 Replies

2. UNIX for Advanced & Expert Users

performance

Hi, 1-in vmstat commande line, in reply, which column is the more important to look and verify if server is very slow ? 2-how can I see how many sessions are opened with the same login ? Many thanks before. (2 Replies)
Discussion started by: big123456
2 Replies

3. UNIX for Advanced & Expert Users

performance

Hi, I have this on a AIX UNIX machine : ps aux| head -20 USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND root 516 23.7 0.0 12 15808 - A 19:38:15 903:13 wait root 774 23.7 0.0 12 15808 - A 19:38:15 902:13 wait root 1290 23.6 0.0 ... (2 Replies)
Discussion started by: big123456
2 Replies

4. UNIX for Advanced & Expert Users

I/O performance

i want to determine I/O performance of an executable, but iostat dont give correct results because the disk that i am writing to and reading from, are not physical disk of the host machine, instead of these local disks we are using a network storage. is there any standard way in unix to get... (2 Replies)
Discussion started by: gfhgfnhhn
2 Replies

5. News, Links, Events and Announcements

Announcing collectl - new performance linux performance monitor

About 4 years ago I wrote this tool inspired by Rob Urban's collect tool for DEC's Tru64 Unix. What makes this tool as different as collect was in its day is its ability to run at a low overhead and collect tons of stuff. I've expanded the general concept and even include data not available in... (0 Replies)
Discussion started by: MarkSeger
0 Replies

6. Solaris

best way and best performance

Hi all, I have two storadge 3510 Fc .. with 12 disks 146Gb ..total 1752Gb each storadge. I need to use about 1.4 Tb of it. and want RAID1 .. I need 13 mount points .. So question: for best performance and redundjancy how I must do it. create 13 logical drives on each stordge with same size... (1 Reply)
Discussion started by: samar
1 Replies

7. Shell Programming and Scripting

Increase sed performance

I'm using sed to do find and replace. But since the file is huge and i have more than 1000 files to be searched, the script is taking a lot of time. Can somebody help me with a better sed command. Below is the details. Input: 1 1 2 3 3 4 5 5 Here I know the file is sorted. ... (4 Replies)
Discussion started by: gpaulose
4 Replies

8. Shell Programming and Scripting

sed / grep / for statement performance - please help

I'm searching the most effective way of doing the following task, so if someone can either provide a working solution with sed or one totally different but more effective then what I've got so far then please go ahead! The debugme directory has 3 subdirectorys and each of them has one .txt file... (7 Replies)
Discussion started by: TehOne
7 Replies

9. UNIX for Dummies Questions & Answers

Performance analysis sed vs awk

Hi Guys, I've wondered for some time the performance analysis between using sed and awk. say i want to print lines from a very large file. For ex say a file with 100,000 records. i want to print the lines 25,000 to 26,000 i can do so by the following commands: sed -n '25000,26000 p'... (11 Replies)
Discussion started by: Irishboy24
11 Replies
NWBPSET(1)							      nwbpset								NWBPSET(1)

NAME
nwbpset - Create a bindery property or set its value SYNOPSIS
nwbpset [ -h ] [ -S server ] [ -U user name ] [ -P password | -n ] [ -C ] DESCRIPTION
nwbpset Reads a property specification from the standard input and creates and sets the corresponding property. The format is determined by the output of 'nwbpvalues -c'. nwbpset will hopefully become an important part of the bindery management suite of ncpfs, together with 'nwbpvalues -c'. See util/nwbpsecurity for an example. As another example, look at the following command line: nwbpvalues -t 1 -o supervisor -p user_defaults -c | sed '2s/.*/ME/'| sed '3s/.*/LOGIN_CONTROL/'| nwbpset With this command, the property user_defaults of the user object 'supervisor' is copied into the property login_control of the user object 'me'. nwbpvalues -t 1 -o me -p login_control -c | sed '9s/.*/ff/'| nwbpset This command disables the user object me. Feel free to contribute other examples! nwbpset looks up the file $HOME/.nwclient to find a file server, a user name and possibly a password. See nwclient(5) for more information. Please note that the access permissions of $HOME/.nwclient MUST be 600 for security reasons. OPTIONS
-h -h is used to print out a short help text. -S server server is the name of the server you want to use. -U user user is the user name to use for login. -P password password is the password to use for login. If neither -n nor -P are given, and the user has no open connection to the server, nwbpset prompts for a password. -n -n should be given if no password is required for the login. -C By default, passwords are converted to uppercase before they are sent to the server, because most servers require this. You can turn off this conversion by -C. AUTHORS
nwbpset was written by Volker Lendecke. See the Changes file of ncpfs for other contributors. nwbpset 8/7/1996 NWBPSET(1)
All times are GMT -4. The time now is 07:45 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy