performence is bad


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers performence is bad
# 1  
Old 06-13-2005
Need assistance to improve performence

Hello experts,

I have a series issue in script that result with bad peformence and I wonder if you can assist me.

For example
I have two files:

File-New, size 15Mb.
File-Old, size 1Mb.

File-New content:
a
b
c
d
File-Old content:
d
f
a
b
c
Result fiel should be:
!a
!b
!c
!d
-f

The current script comapre the lines between these two files in a special order and it takes so long.

The script that currently runs, do the following:
#!/bin/sh
#Input:
# $1 means File-Old
# $2 means File-New
#! - MarkForChange
#- MarkForDeletetion
while read line
do
grep -Fi "$line" $2 >/dev/null 2>&1
if [ "$?" -eq "0" ]; then
echo "! $line"
else
echo "- $line"
fi
done < $1


Do you have any idea how to improve the process.(It take more then hour to run...:-(

Thanks,
Roy.

Last edited by roybe; 06-13-2005 at 12:56 PM.. Reason: More datd
# 2  
Old 06-13-2005
Any idea to improve the performnce ?

I must search every line (one of 1000) in 15-20Mb file and mark ! if it was found or - if doesn't.

The script runs very slow.

Any idea?
# 3  
Old 06-13-2005
you might want to look into using 'comm'

And pls don't multipost and start different threads.
Read the rules before posting.
# 4  
Old 06-13-2005
Thanks

Sorry for thread stuff Smilie

I just change the grep -Fi to fgrep and it is fast man...

I still don't undersatnd why is so different on performence...
# 5  
Old 07-11-2005
Question

Shouldn't be any different fgrep is just script that runs grep -F:

#!/bin/sh
exec grep -F ${1+"$@"}
# 6  
Old 07-11-2005
Quote:
Originally Posted by supaphat21
Shouldn't be any different fgrep is just script that runs grep -F:

#!/bin/sh
exec grep -F ${1+"$@"}
not on Sun/Solaris it ain't!

BTW,
Quote:
Originally Posted by man fgrep
DESCRIPTION
The fgrep (fast grep ) utility searches files for a charac-
ter string and prints all lines that contain that string.
fgrep is different from grep(1) and egrep(1) because it
searches for a string, instead of searching for a
pattern that matches an expression. It
uses a fast and compact algorithm
.
# 7  
Old 07-11-2005
Posix extended grep and the extentions incude -E and -F that duplicate the old egrep and fgrep. On Solaris, this behavior can be obtained by using /usr/xpg4/bin/grep. But read the man page carefully, the old grep has a few things that the new grep doesn't.

And yes, "grep -F" and fgrep should work the same wherever "grep -F" is supported.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bad substitution

Cant undestand :) why i have an error on line 2.it is working on my other boxes #!/bin/bash ret=$(echo Q | timeout 5 openssl s_client connect "${1`hostname`}:${2-443}" -ssl3 2> /dev/null) if echo "${ret}" | grep -q 'Protocol.*SSLv3'; then if echo "${ret}" | grep -q 'Cipher.*0000'; then ... (7 Replies)
Discussion started by: kenshinhimura
7 Replies

2. Solaris

bad options(s)

Hi Folks, I am seeing an issue in solaris, while running a script.. Script contents #!/usr/bin/bash echo $HOME echo "********** RUNNING props on test ********" . /opt/home/weblogic/.profile > /dev/null echo "****** CURRENT BRANCH: $BRANCH *********"; .... (3 Replies)
Discussion started by: ganga.dharan
3 Replies

3. Shell Programming and Scripting

Why I get bad bad substitution when using eval?

Why I get bad replace when using eval? $ map0=( "0" "0000" "0") $ i=0 $ eval echo \${map$i} 0000 $ a=`eval echo \${map$i}` !!!error happens!!! bash: ${map$i}: bad substitution How to resolve it ? Thanks! (5 Replies)
Discussion started by: 915086731
5 Replies

4. Shell Programming and Scripting

Bad Interpreter

Hi. My name is Caleb (a.k.a RagingNinja) form the whited00r forums. (Whited00r makes custom firmware for iOS devices). I have been learning and creating simple shells scripts. I have been recently using VIM for Windows or using VirtualBox to run the UBUNTU OS within VirtualBox to create my shell... (2 Replies)
Discussion started by: RagingNinja
2 Replies

5. UNIX for Advanced & Expert Users

Bad emulation

I am using mpc8xx processor with linux-2.4.20 kernel. These prints are getting flooded to my console almost consistenly. 14 06:29:17 MC4L-TP01 syslog.warn klogd: pte @ 0x10113d88: (0xc51cc100)->(0xc4bd144c)->0x01eec889 Oct 14 06:29:17 MC4L-TP01 syslog.warn klogd: RPN: 01eec PP: 2 SPS: 1 SH:... (1 Reply)
Discussion started by: subratasaharia
1 Replies

6. UNIX for Dummies Questions & Answers

Unix Performence

I have writing quite a few shell scripts to always play with the files. But recently there were some job demand as to write some script to convert a report into a pipe delimited feed file and the volume of the data in it is going to be around some million of records.:mad: 1) Can some help me... (0 Replies)
Discussion started by: nageswarag
0 Replies

7. Shell Programming and Scripting

Script to search a bad record in a file then put the record in the bad file

I need to write a script that can find a bad record (for example: there is date field colom but value provided in the file for this field is N/A) then script shoud searches this pattern and then insert the whole record into the bad file. Example: File1 Name designation dateOfJoining... (2 Replies)
Discussion started by: shilendrajadon
2 Replies

8. UNIX for Advanced & Expert Users

Script to search a bad record in a file then put the record in the bad file

I need to write a script that can find a bad record (for example: there is date field colom but value provided in the file for this field is N/A) then script shoud searches this pattern and then insert the whole record into the bad file. Example: File1 Name designation dateOfJoining... (1 Reply)
Discussion started by: shilendrajadon
1 Replies

9. UNIX for Advanced & Expert Users

System performence.

I am using RHEL AS3. I want to improve the system performence. For that I have disabled the un-used virtual consoles , And I increased the swap space., I disabled the unwanted services which are running background. Can any one tel me other techniques to improve the system performance? Help... (5 Replies)
Discussion started by: praveen_b744
5 Replies

10. UNIX for Dummies Questions & Answers

I just did something bad....

I ran rm -rf lib*, I intended libm*, it deleted the whole lib dir from root. This is a dedicated webserver. What should I do? (4 Replies)
Discussion started by: Lss1
4 Replies
Login or Register to Ask a Question