Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Performance problem in Shell Script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Performance problem in Shell Script
# 8  
Old 08-19-2015
from man grep:

Code:
GREP(1)                     General Commands Manual                    GREP(1)



NAME
       grep, egrep, fgrep - print lines matching a pattern

SYNOPSIS
       grep [OPTIONS] PATTERN [FILE...]
       grep [OPTIONS] [-e PATTERN | -f FILE] [FILE...]

...

       -F, --fixed-strings
              Interpret  PATTERN  as  a  list  of  fixed strings, separated by
              newlines, any of which is to be matched.

...

       -f FILE, --file=FILE
              Obtain patterns  from  FILE,  one  per  line.   The  empty  file
              contains zero patterns, and therefore matches nothing.

...

       -v, --invert-match
              Invert the sense of matching, to select non-matching lines.

# 9  
Old 08-19-2015
grep -F or fgrep:
Code:
fgrep -xvf file2 file1

For an unknown reason (hash implementation?) awk is faster than grep
Code:
awk 'NR==FNR {s[$0]; next} !($0 in s)' file2 file1

This User Gave Thanks to MadeInGermany For This Post:
rbatte1
# 10  
Old 08-19-2015
Quote:
Originally Posted by MadeInGermany
grep -F or fgrep:
Code:
fgrep -xvf file2 file1

For an unknown reason (hash implementation?) awk is faster than grep
Code:
awk 'NR==FNR {s[$0]; next} !($0 in s)' file2 file1

For 700K records, either will do. Handy as awk is, using grep doesn't require one to learn an entire new programming language
These 2 Users Gave Thanks to Corona688 For This Post:
drl rbatte1
# 11  
Old 08-19-2015
i think the above grep command compares 1st line with first line and second line with second line and third line with third line in both the files but I need to compare 1st line of the first file with all the lines in the second file and second line of the first file with all the lines in the second file respectively and print only the lines in the first file that are not matched with the second file.
Thanks for your help on this.
# 12  
Old 08-19-2015
You are mistaken - it tries to match every pattern (or entire line in case you use fgrep) in the pattern file- the one supplied to the -f option - to every line in the the files you present as "targets". But - you must make sure that the patterns are formed in a way that they can be matched in the target files. Here e.g. DOS line terminators can be a killer!
This User Gave Thanks to RudiC For This Post:
rbatte1
# 13  
Old 08-21-2015
@MadeinGermany
Thank you.
AWK command worked like charm.
Is there any links where i can get understand these ?

---------- Post updated at 01:44 PM ---------- Previous update was at 01:43 PM ----------

@RudiC.
Thanks for the explanation.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Performance Issue - Shell Script

Hi, I am beginner in shell scripting. I have written a script to parse file(s) having large number of lines each having multiple comma separated strings. But it seems like script is very slow. It took more than 30mins to parse a file with size 120MB (523564 lines), below is the script code ... (4 Replies)
Discussion started by: imrandec85
4 Replies

2. Shell Programming and Scripting

Performance issue in shell script

Hi All, I am facing performance issue while rinning the LINUX shell script. I have file1 and file 2. File one is the source file and file 2 is lookup file. Need to replace if the pattern is matching in file1 with file2. The order of lookup file is important as if any match then exit... (8 Replies)
Discussion started by: ureddy
8 Replies

3. Shell Programming and Scripting

Shell performance problem with locating scripts

I am currently trying out MKS Toolkit C Shell, and I've no problems with it until I try add directories to PATH that are located on a network drive. When I do that, the shell performance slows down significantly and no longer runs fast. In fact, it takes seconds for something that should take... (1 Reply)
Discussion started by: vas28r13
1 Replies

4. Shell Programming and Scripting

Improve the performance of a shell script

Hi Friends, I wrote the below shell script to generate a report on alert messages recieved on a day. But i for processing around 4500 lines (alerts) the script is taking aorund 30 minutes to process. Please help me to make it faster and improve the performace of the script. i would be very... (10 Replies)
Discussion started by: apsprabhu
10 Replies

5. Shell Programming and Scripting

performance of shell script ( grep command)

Hi, I have to find out the run time for 40-45 different componets. These components writes in to a genreric log file in a single directory. eg. directory is LOG and the log file name format is generic_log_<process_id>_<date YY_MM_DD_HH_MM_SS>.log i am taking the run time using the time... (3 Replies)
Discussion started by: vikash_k
3 Replies

6. Shell Programming and Scripting

Script Performance problem . urgent frnds

HI frnds I have one flat with data and am loading the data into oracle table. While loading , rejected records are captured in log file. Now I want to read the log file and get the all rejected records and the reason for the rejection. I developed the script . its finding 5000 rejected... (7 Replies)
Discussion started by: Gopal_Engg
7 Replies

7. UNIX for Advanced & Expert Users

FTP-Shell Script-Performance issue

Hello All, Request any one of Unix/Linux masters to clarify on the below. How far it is feasible to open a new ftp connection for transferring each file when there are multiple files to be sent. I have developed shell script to send all files at single stretch but some how it doesnt suit to... (3 Replies)
Discussion started by: RSC1985
3 Replies

8. Shell Programming and Scripting

Performance problem with my script ...suggestions pls

Hi , I have included my script below, pls read thro this req. I want my script to run for every hour , the problem is I CANNOT USE CRONTAB which is prohibited inside the company. My script does what it is supposed to do (to determine the memory and then send a email if it crosses a certain... (2 Replies)
Discussion started by: vivsiv
2 Replies

9. Shell Programming and Scripting

shell script performance issues --Urgent

I need help in awk please help immediatly. This below function is taking lot of time Please help me to fine tune it so that it runs faster. The file count is around 3million records # Process Body processbody() { #set -x while read line do ... (18 Replies)
Discussion started by: icefish
18 Replies

10. UNIX for Advanced & Expert Users

Performance of a shell script

Hiii, I wrote a shell script for testing purpose. I have to test around 200thousand entries with the script.When i am doing only for 6000 entries its taking almost 1hour.If i test the whole testingdata it will take huge amount of time. I just want to know is it something dependent on the... (2 Replies)
Discussion started by: namishtiwari
2 Replies
Login or Register to Ask a Question