Visit Our UNIX and Linux User Community


Increase sed performance


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Increase sed performance
# 1  
Old 10-01-2009
Increase sed performance

I'm using sed to do find and replace. But since the file is huge and i have more than 1000 files to be searched, the script is taking a lot of time. Can somebody help me with a better sed command. Below is the details.
Input:

Code:
1
1
2
3
3
4
5
5

Here I know the file is sorted.

My cross reference file will be
Code:
1A
2B
3C
4D
5E
6F

here also the file will be sorted on the first character.

The output I want is
Code:
1A
1A
2B
3C
3C
4D
5E
5E

# 2  
Old 10-01-2009
Code:
 awk 'NR==FNR{a[substr($0,1,1)]=substr($0,2,1);next}a[$0]{printf "%s%s\n",$0,a[$0]}' file2 file1



---------- Post updated at 06:05 PM ---------- Previous update was at 05:59 PM ----------

And if your awk version accepts empty FS:
Code:
 awk -v FS="" -v OFS="" 'NR==FNR{a[$1]=$2;next}a[$0]{print $0,a[$0]}' file2 file1

# 3  
Old 10-01-2009
Just a hint:
If you experience performance problems then you should probably forget about shell at this point and use some programming language. I/O can be far superior to what you can get with shell scripts.
# 4  
Old 10-04-2009
awk command needed

Hi

I need further help on the issue posted by gpaulose. I need awk command to append a string at the end of the lines in a text file where a certain string match is found. I am using grep to find a match in the file and then appending.
I am using sed command at present. But the file size is huge and there are 100s of such files. So sed is taking too much time.

Can someone tell me the awk command to append a string at the end of line where a certain string match s found. Performance is very important here.

Thanks
# 5  
Old 10-04-2009
Quote:
Originally Posted by diksha2207
Hi

I need further help on the issue posted by gpaulose. I need awk command to append a string at the end of the lines in a text file where a certain string match is found. I am using grep to find a match in the file and then appending.
I am using sed command at present. But the file size is huge and there are 100s of such files. So sed is taking too much time.

Can someone tell me the awk command to append a string at the end of line where a certain string match s found. Performance is very important here.

Thanks
Welcome to the forums ! Smilie

Please read the rules of the forum.
This should have been a new post.

Please post us what you had tried so far.

Previous Thread | Next Thread
Test Your Knowledge in Computers #120
Difficulty: Easy
The UNIX certification mark is commercially licensed by AT&T.
True or False?

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Increase the performance of find command.

I'm trying to exclude 'BACKUP', 'STORE', 'LOGGER' folders while searching for all files under a directory "/tmp/moht" Once a file is found I wish to display the filename, the size of the file & the cksum value. Below is the command, I'm using: /opt/freeware/bin/find /tmp/moht -type d -name... (6 Replies)
Discussion started by: mohtashims
6 Replies

2. Shell Programming and Scripting

Using = with sed to increase sequence count

I have a fasta file like this one: >ID1 AAAAAA >ID2 TTTTTT And I am using this sed script to increase the count sequence sed '/^>/s/.*//;/^$/=;/^$/d' text.txt | sed 's/.*/echo ">seq" \$(( ( & + 1 )\/2 ))/e' I get the desired output: >seq 1 AAAAAA >seq 2 TTTTTT However, this... (9 Replies)
Discussion started by: Xterra
9 Replies

3. UNIX for Dummies Questions & Answers

Performance analysis sed vs awk

Hi Guys, I've wondered for some time the performance analysis between using sed and awk. say i want to print lines from a very large file. For ex say a file with 100,000 records. i want to print the lines 25,000 to 26,000 i can do so by the following commands: sed -n '25000,26000 p'... (11 Replies)
Discussion started by: Irishboy24
11 Replies

4. Shell Programming and Scripting

sed / grep / for statement performance - please help

I'm searching the most effective way of doing the following task, so if someone can either provide a working solution with sed or one totally different but more effective then what I've got so far then please go ahead! The debugme directory has 3 subdirectorys and each of them has one .txt file... (7 Replies)
Discussion started by: TehOne
7 Replies

5. AIX

How can I increase a PV?

Hi, I have a 10GB iSCSI LUN attached to an AIX 5.3 system. I increased the LUN to 15GB, but the system is still showing 10GB as the Total Size. How can I get the OS to see the extra space? Do I have to reboot the system? (2 Replies)
Discussion started by: bbbngowc
2 Replies

6. Shell Programming and Scripting

Increase Performance

I have written a code using AWK & sed to compare two files. The structure of the files is like this" Format is this: <bit code> <file code> <string> Follwoed by any numbers of properties lines whic start with a "space" 10101010101111101 XX abcd a AS sasa BS kkk 1110000101010110 XX... (1 Reply)
Discussion started by: sandeep_hi
1 Replies

7. UNIX for Advanced & Expert Users

sed performance

hello experts, i am trying to replace a line in a 100+mb text file. the structure is similar to the passwd file, id:value1:value2 and so on. using the sed command sed -i 's/\(123\):\(\{1,\}\):/\1:bar:/' data.txt works nicely, the line "123:foo:" is replaced by "123:bar:". however, it takes... (7 Replies)
Discussion started by: f3k
7 Replies

8. News, Links, Events and Announcements

Announcing collectl - new performance linux performance monitor

About 4 years ago I wrote this tool inspired by Rob Urban's collect tool for DEC's Tru64 Unix. What makes this tool as different as collect was in its day is its ability to run at a low overhead and collect tons of stuff. I've expanded the general concept and even include data not available in... (0 Replies)
Discussion started by: MarkSeger
0 Replies

9. Solaris

how to increase fs

hi, i installed solaris 9 on my v240 server on 36gb disk. here are the ouputs of the df -h command: # df -h Filesystem size used avail capacity Mounted on /dev/dsk/c1t0d0s0 9.6G 3.4G 6.1G 36% / /proc 0K 0K 0K 0% /proc mnttab ... (6 Replies)
Discussion started by: xuc_xich_duc
6 Replies

Featured Tech Videos