Performance problem in Shell Script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Performance problem in Shell Script
# 1  
Old 08-18-2015
Performance problem in Shell Script

Hi,
I am Shell script beginner.
I wrote a shell programming that will take each line of a file1 and search for it in another file2 and give me the output of the lines that do not exist in the file2.
I wrote it using do while nested loop but the problem here is its running for ever . Is there a way I can improve the performance of the script.
Both of my files contains 700K records each.
# 2  
Old 08-18-2015
Hi sakthisivi, welcome to the forums.

To enhance your script, please share what you have tried so far.

Greetings
# 3  
Old 08-18-2015
Hi,
Quote:
Originally Posted by sakthisivi
I wrote a shell programming that will take each line of a file1 and search for it in another file2 and give me the output of the lines that do not exist in the file2.
no need of a shell script to do such a thing, grep can do it, alone.
# 4  
Old 08-19-2015
And, share small but representative samples of the files.
# 5  
Old 08-19-2015
[edit] I answered the wrong thread: Smilie

Last edited by Corona688; 08-19-2015 at 01:18 PM..
# 6  
Old 08-19-2015
@daPeach

can you tell me how can i do it with a grep command?
# 7  
Old 08-19-2015
Typically, one would use:
Code:
grep -vxFf file2 file1

or try awk:
Code:
awk 'NR==FNR{A[$0]; next} !($0 in A)' file2 file1

These two approaches only work if the lines are exactly the same, with no leading or trailing whilespace in one file, that is missing in the other...


--
Otherwise you could try this adaptation of the awk approach:
Code:
awk '{p=$0; $1=$1} NR==FNR{A[$0]; next} !($0 in A){print p}' file2 file1


--
On Solaris use /usr/xpg4/bin/grep and /usr/xpg4/bin/awk

Last edited by Scrutinizer; 08-19-2015 at 01:53 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Performance Issue - Shell Script

Hi, I am beginner in shell scripting. I have written a script to parse file(s) having large number of lines each having multiple comma separated strings. But it seems like script is very slow. It took more than 30mins to parse a file with size 120MB (523564 lines), below is the script code ... (4 Replies)
Discussion started by: imrandec85
4 Replies

2. Shell Programming and Scripting

Performance issue in shell script

Hi All, I am facing performance issue while rinning the LINUX shell script. I have file1 and file 2. File one is the source file and file 2 is lookup file. Need to replace if the pattern is matching in file1 with file2. The order of lookup file is important as if any match then exit... (8 Replies)
Discussion started by: ureddy
8 Replies

3. Shell Programming and Scripting

Shell performance problem with locating scripts

I am currently trying out MKS Toolkit C Shell, and I've no problems with it until I try add directories to PATH that are located on a network drive. When I do that, the shell performance slows down significantly and no longer runs fast. In fact, it takes seconds for something that should take... (1 Reply)
Discussion started by: vas28r13
1 Replies

4. Shell Programming and Scripting

Improve the performance of a shell script

Hi Friends, I wrote the below shell script to generate a report on alert messages recieved on a day. But i for processing around 4500 lines (alerts) the script is taking aorund 30 minutes to process. Please help me to make it faster and improve the performace of the script. i would be very... (10 Replies)
Discussion started by: apsprabhu
10 Replies

5. Shell Programming and Scripting

performance of shell script ( grep command)

Hi, I have to find out the run time for 40-45 different componets. These components writes in to a genreric log file in a single directory. eg. directory is LOG and the log file name format is generic_log_<process_id>_<date YY_MM_DD_HH_MM_SS>.log i am taking the run time using the time... (3 Replies)
Discussion started by: vikash_k
3 Replies

6. Shell Programming and Scripting

Script Performance problem . urgent frnds

HI frnds I have one flat with data and am loading the data into oracle table. While loading , rejected records are captured in log file. Now I want to read the log file and get the all rejected records and the reason for the rejection. I developed the script . its finding 5000 rejected... (7 Replies)
Discussion started by: Gopal_Engg
7 Replies

7. UNIX for Advanced & Expert Users

FTP-Shell Script-Performance issue

Hello All, Request any one of Unix/Linux masters to clarify on the below. How far it is feasible to open a new ftp connection for transferring each file when there are multiple files to be sent. I have developed shell script to send all files at single stretch but some how it doesnt suit to... (3 Replies)
Discussion started by: RSC1985
3 Replies

8. Shell Programming and Scripting

Performance problem with my script ...suggestions pls

Hi , I have included my script below, pls read thro this req. I want my script to run for every hour , the problem is I CANNOT USE CRONTAB which is prohibited inside the company. My script does what it is supposed to do (to determine the memory and then send a email if it crosses a certain... (2 Replies)
Discussion started by: vivsiv
2 Replies

9. Shell Programming and Scripting

shell script performance issues --Urgent

I need help in awk please help immediatly. This below function is taking lot of time Please help me to fine tune it so that it runs faster. The file count is around 3million records # Process Body processbody() { #set -x while read line do ... (18 Replies)
Discussion started by: icefish
18 Replies

10. UNIX for Advanced & Expert Users

Performance of a shell script

Hiii, I wrote a shell script for testing purpose. I have to test around 200thousand entries with the script.When i am doing only for 6000 entries its taking almost 1hour.If i test the whole testingdata it will take huge amount of time. I just want to know is it something dependent on the... (2 Replies)
Discussion started by: namishtiwari
2 Replies
Login or Register to Ask a Question