Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Insert value to ORACLE table from sqlldr log

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Insert value to ORACLE table from sqlldr log
# 1  
Old 02-17-2009
Data Grep Info from log file..
This is the sample sqlldr log:

------------------------------------------------------------------------------------------------------------

SQL*Loader: Release 9.2.0.7.0 - Production on Sun Feb 8 23:37:02 2009

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

Control File: /home/oracle9/dba_area/billed_recurr_charges.ctl
Data File: /tmw/oradata12/bill_recc_charges_bp01_semenanjung.data
Bad File: /tmw/oradata18/bad/bill_recc_charges_bp01_semenanjung.data.bad
Discard File: none specified

(Allow all discards)

Number to load: ALL
Number to skip: 0
Errors allowed: 50
Bind array: 64 rows, maximum of 256000 bytes
Continuation: none specified
Path used: Conventional

Table CRISPADM.BILLED_RECURR_CHARGES, loaded from every logical record.
Insert option in effect for this table: APPEND
TRAILING NULLCOLS option in effect

Column Name Position Len Term Encl Datatype
------------------------------ ---------- ----- ---- ---- ---------------------
SERVICE_NUM 1:16 16 CHARACTER
BILL_DATE 17:24 8 DATE YYYYMMDD
NULL if BILL_DATE = 0X3030303030303030(character '00000000')
SEF_CODE 25:28 4 CHARACTER
ACCOUNT_NUM 29:41 13 CHARACTER
BILL_PERIOD 42:43 2 CHARACTER
SERVICE_TYPE 44:46 3 CHARACTER
CHARGE_AMT 47:61 15 CHARACTER
CHARGE_RATE 62:76 15 CHARACTER
CHARGE_QTY 77:91 15 CHARACTER
EQUIPMENT_OWNERSHIP_CODE 92:92 1 CHARACTER
CHARGE_EXEMPTION_IND 93:93 1 CHARACTER
SOURCE CONSTANT
Value is 'CASS'

Record 719882: Rejected - Error on table CRISPADM.BILLED_RECURR_CHARGES.
ORA-01400: cannot insert NULL into ("CRISPADM"."BILLED_RECURR_CHARGES"."SEF_CODE")


Table CRISPADM.BILLED_RECURR_CHARGES:
1722967 Rows successfully loaded.
1 Row not loaded due to data errors.
0 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.


Space allocated for bind array: 8128 bytes(64 rows)
Read buffer bytes: 1048576

Total logical records skipped: 0
Total logical records read: 1722968
Total logical records rejected: 1
Total logical records discarded: 0

Run began on Sun Feb 08 23:37:02 2009
Run ended on Sun Feb 08 23:42:19 2009

Elapsed time was: 00:05:17.27
CPU time was: 00:01:30.26

------------------------------------------------------------------------------------------------------------

So, I need to ouput the bolded info into a table:

Code:
insert into PROCEDURE_TRACKING_LOG
(procedure_name, table_name, rows_inserted, rows_updated, rows_deleted, rows_inserted_of_deletion, 
rows_inserted_after_deletion, rows_rejected, start_time, end_time, elapse_time, remarks, subject_area)
values
(proc_name,table_name,row_insert,NULL,NULL,NULL,
NULL,row_reject,start_time,sysdate,elapsed_time,v_ErrorText,'NETWORK INVENTORY');

whereby
Code:
proc_name = 'the filename which runs the sqlldr'
table_name = 'BILLED_RECURR_CHARGES'
row_insert = 1722967
row_reject = 1
start_time = 'Sun Feb 08 23:37:02 2009'
sysdate = 'Sun Feb 08 23:42:19 2009'
elapsed_time = 317 (after conversion to rounded seconds)
v_ErrorText = 'ORA-01400: cannot insert NULL into ("CRISPADM"."BILLED_RECURR_CHARGES"."SEF_CODE")'


Please help me.

Thank you very much.

All your help are much appreciated.. Image Image Image
Last edited by aimy; 02-17-2009 at 09:33 PM..
# 2  
Old 02-17-2009
Tools
Is this impossible to achieve?? Smilie
# 3  
Old 02-17-2009
Why there are still no responses to this question? Smilie
# 4  
Old 02-17-2009
/Moderative Mode ON

First of all, you shouldn't bump up questions, because it is against the rules.

Second: please notice that this is not a help desk. If you want informed, precise answers to be given within a certain timeframe please consider hiring an UNIX expert.

Third: please bear in mind, that all people answering here are VOLUNTEERS. They might be (in fact they are) most times interested in helping you or someone else but they are not obligated to do so. If they do not feel like it (and, frankly, exhibiting a demanding attitude like you did makes them not feeling like it more likely) they do not have to answer at all.

Fourth: all these boards are about is helping one to help himself. Right now you have shown more effort in getting us to work faster than in solving your problems yourself. Take this as an opportunity to think over (and perhaps rearrange) your priorities.

/Moderative Mode OFF, User Mode ON

There is a simple answer to your problem: probably it could be solved with a finite effort and a little sed-scripting.

Longer answer: the real problem lies perhaps in the analysis of the problem and in formulating a solution which is general and reliable enough to let it run unsupervised. For this there are simply not enough data.

Some questions you will have to answer: you have provided a file with exactly one error. How would a file look like when there are two (any arbitrary number of) errors? How does the file look like when there are no errors at all? How will the file look like when there are different types of errors (say, the load stops midways because of some error - file error, disk full, what else)?

A stable and reliable script would be able to parse all these types of output (in fact it should be able to cope with every possible output) therefor you have to take all these different (?) output formats into account.

So please give more details and maybe we can work out a script that does exactly that or try to write one yourself. Provided that the problems mentioned above can be overcome here is a way to isolate your needed values:

You search for a specific value (for the sake of the example we will use start_time) and you know how the line looks like which contains this value:

Code:
.... any text ....
....
Run began on Sun Feb 08 23:37:02 2009
... some more text ....

You can search for this line easily with sed. Now replace the fixed text part "Run began on" with the variable name you want to assign it to to form a declaration:

Code:
sed -n 's/^Run began on/start_time \=/p'

And you will notice that from your logfile you will get out a single declaration reading:

Code:
start_time = Sun Feb 08 23:37:02 2009

This declaration you could directly use to initialize some variables in your script (source the file written this way in).

I hope this helps.

bakunin
# 5  
Old 02-18-2009
Power
Quote:
Originally Posted by bakunin
/Moderative Mode ON

First of all, you shouldn't bump up questions, because it is against the rules.

Second: please notice that this is not a help desk. If you want informed, precise answers to be given within a certain timeframe please consider hiring an UNIX expert.

Third: please bear in mind, that all people answering here are VOLUNTEERS. They might be (in fact they are) most times interested in helping you or someone else but they are not obligated to do so. If they do not feel like it (and, frankly, exhibiting a demanding attitude like you did makes them not feeling like it more likely) they do not have to answer at all.

Fourth: all these boards are about is helping one to help himself. Right now you have shown more effort in getting us to work faster than in solving your problems yourself. Take this as an opportunity to think over (and perhaps rearrange) your priorities.

/Moderative Mode OFF, User Mode ON

There is a simple answer to your problem: probably it could be solved with a finite effort and a little sed-scripting.

Longer answer: the real problem lies perhaps in the analysis of the problem and in formulating a solution which is general and reliable enough to let it run unsupervised. For this there are simply not enough data.

Some questions you will have to answer: you have provided a file with exactly one error. How would a file look like when there are two (any arbitrary number of) errors? How does the file look like when there are no errors at all? How will the file look like when there are different types of errors (say, the load stops midways because of some error - file error, disk full, what else)?

A stable and reliable script would be able to parse all these types of output (in fact it should be able to cope with every possible output) therefor you have to take all these different (?) output formats into account.

So please give more details and maybe we can work out a script that does exactly that or try to write one yourself. Provided that the problems mentioned above can be overcome here is a way to isolate your needed values:

You search for a specific value (for the sake of the example we will use start_time) and you know how the line looks like which contains this value:

Code:
.... any text ....
....
Run began on Sun Feb 08 23:37:02 2009
... some more text ....

You can search for this line easily with sed. Now replace the fixed text part "Run began on" with the variable name you want to assign it to to form a declaration:

Code:
sed -n 's/^Run began on/start_time \=/p'

And you will notice that from your logfile you will get out a single declaration reading:

Code:
start_time = Sun Feb 08 23:37:02 2009

This declaration you could directly use to initialize some variables in your script (source the file written this way in).

I hope this helps.

bakunin
Thanks for your response and please do forgive me about my bad behavior.. Smilie

It's just that I feel my question is not that complicated but yet not one seems to answer.. Smilie

While waiting for the solution, I take my own iniative to study a bit about awk utility.

So, this is what I got:

Code:
table_name = awk '/Table.+[:]/ {print substr($2,10)}' testfile.log
row_insert = awk '/Rows successfully loaded./ {print $1}' testfile.log
row_reject = awk '/not loaded due to data errors./ {print $1}' testfile.log
v_ErrorText = awk '/^ORA-/ {print}' testfile.log
start_time = awk '/Run began on/ {print $5, $6, $7, $8}' testfile.log
sysdate = '/Run ended on/ {print $5, $6, $7, $8}' testfile.log
elapsed_time = awk '/Elapsed/ {print substr($4,1,2)*3600 + substr($4,4,2)*60 + substr($4,7)}' testfile.log

sqlplus -s crispadm/admcrisp <<EOF
set autocommit on;
set echo on;
set serveroutput on;

insert into PROCEDURE_TRACKING_LOG
(procedure_name, table_name, rows_inserted, rows_updated, rows_deleted, rows_inserted_of_deletion, 
rows_inserted_after_deletion, rows_rejected, start_time, end_time, elapse_time, remarks, subject_area)
values
('TEST',replace($table_name,':'),$row_insert,NULL,NULL,NULL,
NULL,$row_reject,$start_time,$sysdate,round($elapsed_time),$v_ErrorText,'NETWORK INVENTORY');

EOF

All the awk command works fine:
Code:
$ awk '/Table.+[:]/ {print substr($2,10)}' testfile.log
BILLED_RECURR_CHARGES:

$ awk '/Rows successfully loaded./ {print $1}' testfile.log
1722967

$ awk '/not loaded due to data errors./ {print $1}' testfile.log
1

$ awk '/^ORA-/ {print}' testfile.log
ORA-01400: cannot insert NULL into ("CRISPADM"."BILLED_RECURR_CHARGES"."SEF_CODE")

$ awk '/Run began on/ {print $5, $6, $7, $8}' testfile.log
Feb 08 23:37:02 2009

$ awk '/Run ended on/ {print $5, $6, $7, $8}' testfile.log
Feb 08 23:42:19 2009

$ awk '/Elapsed/ {print substr($4,1,2)*3600 + substr($4,4,2)*60 + substr($4,7)}' testfile.log
317.27

So, where i do went wrong in the script to set all that awk commands as a variable?

Thank you.
# 6  
Old 02-18-2009
Quote:
Originally Posted by aimy
It's just that I feel my question is not that complicated but yet not one seems to answer..
One of the reasons questions get answered is the are "interesting" to solve: that means usually: non-trivial, non-obvious, complicated. Its an ironic fact that questions easy to be answered might take a longer time to actually be answered than the complicated and awkward ones.

Quote:
Originally Posted by aimy
While waiting for the solution, I take my own iniative to study a bit about awk utility.
That is exactly the right approach.


Quote:
Originally Posted by aimy
Code:
table_name = awk '/Table.+[:]/ {print substr($2,10)}' testfile.log
... rest snipped...

So, where i do went wrong in the script to set all that awk commands as a variable?
First off, your solution with awk is about the same line as my suggestion of sed - in this regard these two tools do about the same even if they do it in a somewhat different way.

Secondly there is a subtle syntactical error in your code: a variable assignment in shell scripts must not use blanks surrounding the "="-sign. Notice the difference between the two lines:

Code:
x = 100   # wrong
x=100    # correct

The reason is that the shell takes everything up to the "=" to be the variables name, hence when you write "x =" you are assigning a variable named "x " (x followed by a blank) and this is probably not what you intended.

Now for the main problem with your code, which is also of syntactical nature. You do NOT want to assign a string that resembles a command to the variable but you want the output of the command be assigned instead. Therefore you must tell the shell to execute the command separately first and then assign its output. Consider the following lines:

Code:
var=date
var=$(date)

The first line will assign the word "date" to the variable, the second one will first execute the "date"-command (whatever is between $(...) ) and execute it (in a separate shell instance, btw.). Whatever output is produced is then replacing the "$(....)" and only then the commandline is executed assigning the variable.

What does that mean to your problem? Edit the lines in the following way:
Code:
table_name="$(awk '/Table.+[:]/ {print substr($2,10)}' testfile.log)"

PS: some might suggest using backticks (`command`) instead of the $(command) construct. Backticks are an outdated way of basically achieving the same, but should be avoided because the newer construct is much more flexible and also better readable. It is one of the most prominent goals of shell scripting to produce code that is easily maintainable, easily read, easily understood. Backticks look in some fonts so similar to single quotes that because of this alone they should be abandoned. This being not the only disadvantage they have they should be avoided even more.

I hope this helps.

bakunin
Last edited by bakunin; 02-18-2009 at 09:28 AM..
# 7  
Old 02-19-2009
Bug
Quote:
Originally Posted by bakunin
One of the reasons questions get answered is the are "interesting" to solve: that means usually: non-trivial, non-obvious, complicated. Its an ironic fact that questions easy to be answered might take a longer time to actually be answered than the complicated and awkward ones.



That is exactly the right approach.




First off, your solution with awk is about the same line as my suggestion of sed - in this regard these two tools do about the same even if they do it in a somewhat different way.

Secondly there is a subtle syntactical error in your code: a variable assignment in shell scripts must not use blanks surrounding the "="-sign. Notice the difference between the two lines:

Code:
x = 100   # wrong
x=100    # correct

The reason is that the shell takes everything up to the "=" to be the variables name, hence when you write "x =" you are assigning a variable named "x " (x followed by a blank) and this is probably not what you intended.

Now for the main problem with your code, which is also of syntactical nature. You do NOT want to assign a string that resembles a command to the variable but you want the output of the command be assigned instead. Therefore you must tell the shell to execute the command separately first and then assign its output. Consider the following lines:

Code:
var=date
var=$(date)

The first line will assign the word "date" to the variable, the second one will first execute the "date"-command (whatever is between $(...) ) and execute it (in a separate shell instance, btw.). Whatever output is produced is then replacing the "$(....)" and only then the commandline is executed assigning the variable.

What does that mean to your problem? Edit the lines in the following way:
Code:
table_name="$(awk '/Table.+[:]/ {print substr($2,10)}' testfile.log)"

PS: some might suggest using backticks (`command`) instead of the $(command) construct. Backticks are an outdated way of basically achieving the same, but should be avoided because the newer construct is much more flexible and also better readable. It is one of the most prominent goals of shell scripting to produce code that is easily maintainable, easily read, easily understood. Backticks look in some fonts so similar to single quotes that because of this alone they should be abandoned. This being not the only disadvantage they have they should be avoided even more.

I hope this helps.

bakunin
Wow.. I am so impressed with your lengthy explanation to my question.. Smilie

Thank you very much sir.. Image Image Image

I really meant it.

Yeah, I already solved my problem yesterday when I noticed that when I put a space surrounding the '=' sign it doesn't work. But I am not really sure that is the real cause of the problem, so thanks for telling me the fact.

Also, the '`' sign surrounding the command. I used that coz I thought the correct syntax should be both i.e. without a surrounding space between '=' and also have to put that '`' sign. So, once again, thanks a lot.

My question now is about.. If I put this script inside a filename called test_script.sh:
Code:
table_name="$(awk '/Table.+[:]/ {print substr($2,10)}' testfile.log)"
row_insert="$(awk '/Rows successfully loaded./ {print $1}' testfile.log)"
row_reject="$(awk '/not loaded due to data errors./ {print $1}' testfile.log)"
v_ErrorText="$(awk '/^ORA-/ {print}' testfile.log)"
start_time="$(awk '/Run began on/ {print $5, $6, $7, $8}' testfile.log)"
sysdate="$(awk '/Run ended on/ {print $5, $6, $7, $8}' testfile.log)"
elapsed_time="$(awk '/Elapsed/ {print substr($4,1,2)*3600 + substr($4,4,2)*60 + substr($4,7)}' testfile.log)"

sqlplus -s crispadm/admcrisp <<EOF
set autocommit on;
set echo on;
set serveroutput on;

insert into PROCEDURE_TRACKING_LOG
(procedure_name, table_name, rows_inserted, rows_updated, rows_deleted, rows_inserted_of_deletion, rows_inserted_after_deletion, rows_rejected, 
start_time, end_time, elapse_time, remarks, subject_area)
values
('$curfile',replace('$table_name',':'),$row_insert,NULL,NULL,NULL,NULL,$row_reject,to_date('$start_time','MON dd hh24:mi:ss yyyy'), 
to_date('$sysdate','MON dd hh24:mi:ss yyyy'), round($elapsed_time),'$v_ErrorText','NETWORK INVENTORY');

EOF

How do I set that variable called $curfile so that it will be returning the script filename itself i.e. test_script.sh

Can anyone please help me?

Thank you very much.. Image
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script to create the SQLLDR control file from Oracle table.

I need to create the shell script load the few 100 table using the SQLLDR. Need to generate the control file for each table using oracle table. table has "table names" and "column names" using this need create the control file. example table rows below table_nme column_nme DEPT ... (2 Replies)
Discussion started by: pimmit22043
2 Replies

2. UNIX and Linux Applications

Help in copying table structure to another table with constraints in Oracle

hi, i need to copy one table with data into another table, right now am using create table table1 as select * from table2 i want the constraints of table1 to be copied to table2 also , can anyone give me some solution to copy the constraints also, now am using oracle 10.2.0.3.0... (1 Reply)
Discussion started by: senkerth
1 Replies

3. Shell Programming and Scripting

Insert script result into Oracle Table

Hi All, I want to insert STAT and ENDTIME values for each job in joblist into TBL_DAILY_STATUS table. Eg: insert into tbl_daily_status values(STAT,ENDTIME); Please help me on this. #!/bin/ksh joblist="com_abc_job com_abc_dot_job com_abc_seq com_abc_det" for i in $joblist do... (8 Replies)
Discussion started by: vichuelaa
8 Replies

4. Shell Programming and Scripting

Insert into Oracle table thru UNIX - linux 2.6.9-89

Hi, I am trying to insert a record into a table (say dips_tbl) which resides in Oracle DB through a ksh script. I want to insert records into few of the table columns-not all. I'll give an e.g. for the date column "CREATE_DATE". For that I first execute SQL1="SELECT SYSDATE FROM DUAL" ... (1 Reply)
Discussion started by: dips_ag
1 Replies

5. Shell Programming and Scripting

select values from db1 table and insert into table of DB2

Hi I am having three oracle databases running in three different machine. their ip address is different. from one of the DB am able to access both the databases.(means am able to select values and insert values in to tables individually.) I need to fetch some data from DB1 table(say DB1 ip is... (2 Replies)
Discussion started by: aemunathan
2 Replies

6. Shell Programming and Scripting

Check the record count in table (table in oracle)

I have requirement: 1) Check the record count in table (table in oracle) 2) If records exists generate the file for existing records and wait for some time then Go to sleep mode and Again check the record count after 10 min.......... (Loop this process if record count >0). 3) Generate touch... (1 Reply)
Discussion started by: kamineni
1 Replies

7. Shell Programming and Scripting

Check the record count in table (table in oracle)

I have requirement: 1) Check the record count in table (table in oracle) 2) If records exists generate the file for existing records and wait for some time (Go to sleep mode) and Again check the record count after 10 min.......... (Loop this process if record count >0). 3) Generate touch... (1 Reply)
Discussion started by: kamineni
1 Replies

8. UNIX for Advanced & Expert Users

unix script for update or insert records from a file to a oracle table

Hi, I have delimited file(|). Sample data: 1|name|50009|DS24|0|12 2|name|30009|DS24|0|13 3|name|20409|DS24|0|14 4|name|20009|DS24|0|15 5|name|10009|DS24|0|16 I want to load this data into a oracle table (update and insert) Please help me the commands and also... (1 Reply)
Discussion started by: unihp1
1 Replies

9. Programming

How can i load or insert a table in oracle from c language thru unix environment

I'm having a oracle server and i'm having a table in that. I'm having a linux server which is in network with the oracle server. I need to write a c program in linux env when on execution loads the table with the text file given as input. Please explain me the flow of process in that and also... (6 Replies)
Discussion started by: rramprasad
6 Replies
Login or Register to Ask a Question