creating data loading script


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users creating data loading script
# 1  
Old 02-02-2010
creating data loading script

asdfasdfasdfasdf???

Last edited by noorm; 02-02-2010 at 07:15 AM.. Reason: code tags
# 2  
Old 02-02-2010
What are you asking? asdfasdfasdf is not a question.
# 3  
Old 02-02-2010
Jim,

My question is below ---

We have a below script and based on which we have create a script which starts sqlldr loading.

Code:
#!/bin/ksh
# mon_large_files.sh
# Monitors for large files whose BCP out has finished successfully
# and it will then call pgm to run sqlldr for that program
# Calls: DATACITI_orctl_load.sh $FILE_NAME
# where $FILE_NAME is in mon_large_files.dat
# DATACITI_orctl_load.sh must 
#       1. Find correct sqlldr string in DATACITI*_orctl.sh and run it
#       2. If suceessful sends email and comments out entry in DATACITI*_orctl.ch
################################################################################
#export NFS="/DUMPS@uat1"
export NFS="/DUMPS@uat2"
export SCRIPT_DIR="$NFS/dump/scripts"
export LARGEFILE_DAT="$SCRIPT_DIR/mon_large_files.dat"
export LOGDIR="$NFS/sybase_dump/logs"
#LOGFILE name based on FILENAME: strip off dbo piece and dat and add log instead
#Ex: dat=DATACITI_CITI.dbo.trans_data_0000000a_0000089e.dat, log=DATACITI_CITI_trans_data_0000000a_0000089e_bcpOut.log
######################################################################################################################:w
cat $LARGEFILE_DAT | while read DATFILE
do
        #echo DatFile: $DATFILE
        #LOGFILE=`echo $DATFILE| sed s/.dbo./_/g |sed s/".dat"/".log"/g `
        LOGFILE=`echo $DATFILE| nawk '{gsub(".dbo.","_"); $0=substr($0,1,length($0)-4)"_bcpOut.log"; print}'`
        ENDFLAG=`tail $LOGDIR/$LOGFILE| grep -c "Ending bcpz" `
        if [ "$ENDFLAG" -eq 1 ]
        then
                #^If load started already, do not invoke pgm, Check for sqlldr log file or DATACITI_orctl_load.sh does this?
                echo "$LOGFILE indicates completes and load not yet started.."
                echo "\tInvoking [test]: $SCRIPT_DIR/DATACITI_orctl_load.sh $DATFILE"           
        fi
done
#

INFO:
-----
If we run monitor script will call your program(DATACITI_orctl_load.sh): eg: DATACITI_orctl_load.sh $DATFILE

Our program will:
1. Verify that we have not already loaded this file
Code:
#!/bin/bash
lockfile=/tmp/DATACITI_orctl_load.lock
if [ ! -e $lockfile ]; 
then
trap "rm -f $lockfile; 
exit" INT TERM EXIT
touch $lockfile
#call mainprogram
rm $lockfile
trap - INT TERM EXIT
else
echo "DATACITI_orctl_load is already running"

2. Do the load by finding sqlldr command in DATACITI_*_orctl1n.sh

eg: DATACITI*_orctl1n.sh file contents-
---------------------------------
Code:
sqlldr DATACITI_CITI/today05@UAT control=Oracle/dbo_DATACITI_CITI.field_cnt_0000000a_00001d11.ctl direct=true 
columnarrayrows=10000 readsize=104857600 
log=log/dbo_DATACITI_CITI.field_cnt_0000000a_00001d11.log 
bad=log/dbo_DATACITI_CITI.field_cnt_0000000a_00001d11.bad 
discard=log/dbo_DATACITI_CITI.field_cnt_0000000a_00001d11.dis

3. Once successful, prevent this from being loaded again; (may need to comment out entry in DATACITI_*_orctl1n.sh)

Code:
$ cat  mon_large_files.dat
DATACITI_CITI.dbo.trans_data_0000000a_0000089e.dat
DATACITI_CITI.dbo.trans_record.dat
DATACITI_CITI.dbo.field_0000000a_0000089e.dat
DATACITI_CITI.dbo.trans_header.dat
DATACITI_CITI.dbo.field_0000000a_000004b1.dat
DATACITI_CITI.dbo.trans_data_0000000a_00001322.dat
DATACITI_CITI.dbo.trans_data_00000001_00000c09.dat
DATACITI_CITI.dbo.field_00000001_00000c09.dat
DATACITI_CITI.dbo.trans_data_0000000a_000004b1.dat
DATACITI_CITI.dbo.trans_data_0000000a_0000003d.dat



---------- Post updated at 05:00 PM ---------- Previous update was at 04:59 PM ----------

Jim u can change the script in point 1. as well

Last edited by pludi; 02-02-2010 at 07:50 AM.. Reason: code tags, please...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Creating bash script to process a data file based on the menu

Hey, im fairly new to unix and Im trying to make this unix project that would display a menu and do the following. MENU =========================== (p, P) Print users info (a, A) Add new user (s, S) Search user (d, D) Delete user (x,X) Exit Enter your choice: Trying to... (3 Replies)
Discussion started by: ultimaxtrd
3 Replies

2. Shell Programming and Scripting

Creating loops inside a file and extracting and loading data

Help needed (1 Reply)
Discussion started by: Chand Shrestha
1 Replies

3. Shell Programming and Scripting

Looping a flat file and loading data

Hi, I am very new to UNIX environment. And I am stuck in this problem. In my Unix server, I have a file called "tablelist.txt" which gives me a list of the tables in my oracle database like shown below. bmp.table1 bmp.table2 bmp.table3 : : bmp.table20 All these tables have 'batch_id'... (1 Reply)
Discussion started by: Reese coutinho
1 Replies

4. Shell Programming and Scripting

Loading data in oracle using shell scripts

Hi , I have a scenario, i have a directory where i receive around 14-15 files at a interval of 20-40 min not fixed, i want to write a unix scripts which invoke sqlldr command to load files into oracle automatically as soon as the file hit the directory. Any help will be appreciated. ... (4 Replies)
Discussion started by: guddu_12
4 Replies

5. Programming

Loading Data in shared memory (C++)

I'll try to keep this short, but basically I need to figure out a way to load data in shared memory (this file will be called load.c) I will later access the data with a print.c program. The "data" is in the form of a student database that looks like this John Blakeman 111223333 560... (7 Replies)
Discussion started by: Mercfh
7 Replies

6. Shell Programming and Scripting

loading the data to the partitioned table using procedure

Hi one help, I need one procedure to load data from flat file to table. Table name as input parameter for the procedure. can anyone help me Thanks, Raj, (1 Reply)
Discussion started by: easterraj
1 Replies

7. UNIX for Dummies Questions & Answers

data is seen as NULL after loading into database

hello, when I load a data from text file all the values become NULL in the table. Please help me with this problem. Thanks sheen (15 Replies)
Discussion started by: sheen
15 Replies

8. Shell Programming and Scripting

unix script for loading a data from a file into database

Hi All, I am trying to load a data from a files in a particular directory to a database.. cd $SCC isql metdb >> ${LOGDATA}/LOAD.LOG < ! load from "${LDATA}/${FORM}.ld" insert into $LOADTABLE ! But it's showing the error "syntax error at line 46 : `<<' unmatched" Can u plz help me... (5 Replies)
Discussion started by: grajesh_955
5 Replies

9. Shell Programming and Scripting

Problem of Data Loading....

i am studying a script which is used for data loading. it has functions which deletes all the existing data before loading and then loads new fresh data. but i am stuck up at function Replace into table ( col 1,col 2....) Does this signify All Inserts. (1 Reply)
Discussion started by: ankitgupta
1 Replies

10. Shell Programming and Scripting

Shell Script for Data loading in Oracle

Hi All I am new to unix. I need a shell script to load a datafile in to oracle. I already have a control file, and data file. all I need is if i execute the shell it must load the data using the ctl file to table. Control file : PAY0001.ctl Datafile : mon_grs_det.dat log file :... (3 Replies)
Discussion started by: raghuraja_r
3 Replies
Login or Register to Ask a Question