Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Script Performance problem . urgent frnds

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script Performance problem . urgent frnds
# 1  
Old 07-31-2009
Script Performance problem . urgent frnds
HI frnds

I have one flat with data and am loading the data into oracle table. While loading , rejected records are captured in log file. Now I want to read the log file and get the all rejected records and the reason for the rejection.
I developed the script . its finding 5000 rejected records. Am writing the records into one output flat file. But its taking 2 hours to for reading and populating the output file.

I have attached my script here. plz plz help frnd


i have attached my script and input sample file with 2 rejected records
Sample Log file.txt (14.8 KB) 
my script.sh.txt (1.7 KB) 
# 2  
Old 07-31-2009
Does your outer while loop ever finish? You never actually increment $i (or decrement $nr_of_occ).
# 3  
Old 07-31-2009
YEs Am incrementing the things.. let i=$i+1 . i forget to copy that.

plz give me what will be the problem . what wll resolve my problem
# 4  
Old 07-31-2009
Quote:
urgent frnds
These kind of subject line, is not useful at all.

Quote:
grep -n "ORA-" $1 |cut -d: -f1
Two commands are not needed, can be done with just awk

Quote:
sed -n "$rec_start,$rec_end p " $1 > data_block.txt
output to the file, at the end of the while block
# 5  
Old 07-31-2009
Quote:
Originally Posted by matrixmadhan
These kind of subject line, is not useful at all.


Two commands are not needed, can be done with just awk


output to the file, at the end of the while block
Code:
 
hi matrixmadha,
actaull that allso part of the req. but hvnt include all those thing. i am facing problem only with writting out_file.txt. it take much time frnd :(
Plz .. i hope as u r a experienced , u can resolve my problem

# 6  
Old 07-31-2009
A solution with awk :
Code:
# ora.sh

awk -v Outfile=ora.txt '

/^ORA-/ {
   split($0, f, "\"");
   Target_schema = f[2];
   Target_table  = f[4];
   Ora_error     = $0;
}

/Targ Rowid=/ { 
   Rejected_count++ ;
   Get_detail++;
   printf "#%d => %s\n", Rejected_count, Ora_error;
   next;
}

/^)/  && Get_detail {
   print Rejected_count,Values_detail > Outfile;
   Values_detail = "";
   Get_detail = 0;
   next;
}

Get_detail {
   sub(/\(.*\):/,"=");
   Values_detail = Values_detail $0 "|"
}

END {
   print "Total Lines in Session Log",FILENAME,"is :",NR;
   print "Total Nr of Rejected Records Captured in Sesslog is :",Rejected_count;
} 
' ora.log

Input file is ora.log and result file is ora.txt
The script takes 36 sec for 16384 rejected records
Code:
$ grep -c 'Targ Rowid' ora.log
16384
$ time ./ora.sh | tail -10

real   35.86
user   34.63
sys    0.71
#16377 => ORA-2222: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
#16378 => ORA-333: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
#16379 => ORA-2222: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
#16380 => ORA-333: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
#16381 => ORA-2222: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
#16382 => ORA-333: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
#16383 => ORA-2222: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
#16384 => ORA-333: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
Total Lines in Session Log ora.log is : 1884160
Total Nr of Rejected Records Captured in Sesslog is : 16384
$ head -10 ora.txt | cut -c1-100
1 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE0222202"|SMA
2 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE1111"|SMA_MA
3 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE0222202"|SMA
4 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE1111"|SMA_MA
5 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE0222202"|SMA
6 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE1111"|SMA_MA
7 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE0222202"|SMA
8 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE1111"|SMA_MA
9 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE0222202"|SMA
10 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE1111"|SMA_M
$

jean-Pierre.
# 7  
Old 08-03-2009
Quote:
Originally Posted by aigles
A solution with awk :
Code:
# ora.sh
 
awk -v Outfile=ora.txt '
 
/^ORA-/ {
   split($0, f, "\"");
   Target_schema = f[2];
   Target_table  = f[4];
   Ora_error     = $0;
}
 
/Targ Rowid=/ { 
   Rejected_count++ ;
   Get_detail++;
   printf "#%d => %s\n", Rejected_count, Ora_error;
   next;
}
 
/^)/  && Get_detail {
   print Rejected_count,Values_detail > Outfile;
   Values_detail = "";
   Get_detail = 0;
   next;
}
 
Get_detail {
   sub(/\(.*\):/,"=");
   Values_detail = Values_detail $0 "|"
}
 
END {
   print "Total Lines in Session Log",FILENAME,"is :",NR;
   print "Total Nr of Rejected Records Captured in Sesslog is :",Rejected_count;
} 
' ora.log

Input file is ora.log and result file is ora.txt
The script takes 36 sec for 16384 rejected records
Code:
$ grep -c 'Targ Rowid' ora.log
16384
$ time ./ora.sh | tail -10
 
real   35.86
user   34.63
sys    0.71
#16377 => ORA-2222: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
#16378 => ORA-333: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
#16379 => ORA-2222: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
#16380 => ORA-333: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
#16381 => ORA-2222: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
#16382 => ORA-333: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
#16383 => ORA-2222: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
#16384 => ORA-333: value too large for column "STN"."STN_SAP_MATERIAL_ATTR"."SMA_STD_DESCR" (actual: 19, maximum: 18)
Total Lines in Session Log ora.log is : 1884160
Total Nr of Rejected Records Captured in Sesslog is : 16384
$ head -10 ora.txt | cut -c1-100
1 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE0222202"|SMA
2 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE1111"|SMA_MA
3 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE0222202"|SMA
4 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE1111"|SMA_MA
5 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE0222202"|SMA
6 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE1111"|SMA_MA
7 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE0222202"|SMA
8 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE1111"|SMA_MA
9 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE0222202"|SMA
10 SMA_FEED_RECEIVED_ID = "MASTER_MATERIAL_ATTR_222222"|SMA_SOURCE_SYSTEM_INSTANCE = "BRE1111"|SMA_M
$

jean-Pierre.

Code:
 
Dear jean-Pierre,
 
 Thanks for your reply. i will try the solution today.
before that , can u plz tell me the problem with code that i developed.
what causes the performance problem. plz tel me . 
how you could rectify that.?
whats the special with AWK utility ?
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

URGENT Reading a file and assessing the syntax shell script URGENT

I am trying to write a shell script which takes an input file as an arguement in the terminal e.g. bash shellscriptname.sh input.txt. I would like for the file to be read line by line each time checking if the .txt file contains certain words or letters(validating the syntax). If the line being... (1 Reply)
Discussion started by: Gurdza32
1 Replies

2. Shell Programming and Scripting

Performance problem in Shell Script

Hi, I am Shell script beginner. I wrote a shell programming that will take each line of a file1 and search for it in another file2 and give me the output of the lines that do not exist in the file2. I wrote it using do while nested loop but the problem here is its running for ever . Is there... (12 Replies)
Discussion started by: sakthisivi
12 Replies

3. UNIX for Advanced & Expert Users

Performance problem with bidirectional nc

Working on a simple, half duplex network diagnostic that will run anywhere using nc and dd. Performance is symmetrical with sink and source nc processes open as a server: nc -vkl 5000 > /dev/null & cat /dev/zero | nc -vkl 5001 & With this on the client: nc host0 5001 | dd of=/dev/null... (0 Replies)
Discussion started by: netdrx
0 Replies

4. Solaris

Performance problem

Hi All, There is a virtual user "ecoouk" which logs on to the server and runs some scripts. I want to know how much server performance can I gain if I put off all the scripts run by this user. Please tell me how to analyse how much resources a specific user is using. Regards, Abhishek (3 Replies)
Discussion started by: max29583
3 Replies

5. Shell Programming and Scripting

Getting Following problem when opening shell script (Very Urgent)

Hi All, Actually when i am trying to open my shell script file 1.sh then i am getting the following error. > vi 1.sh "/var/tmp/ExdNaarK" No space left on device Can anybody tell me that how to rectify it. It is very urgent. Because i am not able to do any work due to the above error. ... (1 Reply)
Discussion started by: sunitachoudhury
1 Replies

6. Shell Programming and Scripting

Performance problem with my script ...suggestions pls

Hi , I have included my script below, pls read thro this req. I want my script to run for every hour , the problem is I CANNOT USE CRONTAB which is prohibited inside the company. My script does what it is supposed to do (to determine the memory and then send a email if it crosses a certain... (2 Replies)
Discussion started by: vivsiv
2 Replies

7. Shell Programming and Scripting

shell script performance issues --Urgent

I need help in awk please help immediatly. This below function is taking lot of time Please help me to fine tune it so that it runs faster. The file count is around 3million records # Process Body processbody() { #set -x while read line do ... (18 Replies)
Discussion started by: icefish
18 Replies

8. Shell Programming and Scripting

urgent: script problem???

#!/usr/bin/sh read file #exec 3<&0 exec 0<$file while read line do if then BOUND=$line break fi count=`expr $count + 1` done #exec 0<&3 echo $BOUND sh check.sh (4 Replies)
Discussion started by: skyineyes
4 Replies

9. UNIX for Advanced & Expert Users

performance problem

Hello, I have a mail server (sendmail) with SUNOS 5.5.1. Just recently it began to respond very slowly. I used vmstat to check the performance data. Only interupt, system call and CPU context swiching are relatively high. Other statistics are normal, especially CPU utilization are very... (5 Replies)
Discussion started by: caoai
5 Replies
Login or Register to Ask a Question