split process files in parallel and rejoin - Shell Programming and Scripting

xiamin · #1 01-10-2006

Hi

I need to split a huge file into multiple smaller files using split command.

After that i need to process each file in the back ground with sql loader .Sql loader is a utlity to load CSV files into oracle .

Check the status of each of these sqlloaders and then after sucessfull completion join the file back again.

Any idea on how to do run the process in background and then check for their sucessful execution

regards
Hrishy

mahendramahendr · #2 01-10-2006

Script :
======

#!/usr/bin/ksh

split -l 1000 hugefile.txt -a hugefile

for fname in hugefile*
do
sqlldr user/password control=xyz.ctl log=${fname}.log data=$fname &
done

echo "Waiting for background process to complete...."

wait

echo "All files loaded..!!"

For finding the status, you can simply grep all the log files or count the lines in bad file.... not sure whether there is a way to trap directly from sqlldr command..

The above split will not actually split the original file, hence no need to join the splitted files back... simply delete the splitted files.

I have not tested the above stuff, correct if you have any syntax errors..

xiamin · #3 01-11-2006

Hi Mahendar

You almost compeletd the assignement for me :-).Thank you very much.

What i am still wundering is that how do i check each background process is there any away say something like

job -l and then look for done command if it fails then it will have something else for the return code.How di build this into the script.

regards
Hrishy

vino · #4 01-11-2006

Quote:

Originally Posted by xiamin

You almost compeletd the assignement for me :-).Thank you very much.

School/University assignment ?

If it is take a look at point 6 of the rules

(6) Do not post classroom or homework problems.

xiamin · #5 01-11-2006

Hi Vino

Nope its not a school assignement problem :-).I am sorry if my wordings sounded like that.

I knwo that we can submit the jobs in background using the & commnad but how do we monitor each job ?

something like job -l and then what how do i know if it sucesfully completed or not if it failed which is the one that failed.

regards
Hrishy

jim mcnamara · #6 01-11-2006

the {$fname}.log files will each have a section like this at the end of the file:

Code:

Table UABBDBT:
  1 Row successfully loaded.
  0 Rows not loaded due to data errors.
  0 Rows not loaded because all WHEN clauses were failed.
  0 Rows not loaded because all fields were null.


Space allocated for bind array:                 255420 bytes(55 rows)
Read   buffer bytes: 1048576

Total logical records skipped:          0
Total logical records read:             1
Total logical records rejected:         0
Total logical records discarded:        0

Run began on Tue Jul 19 11:28:38 2005
Run ended on Tue Jul 19 11:28:39 2005

Elapsed time was:     00:00:00.45
CPU time was:         00:00:00.14

grep for "not loaded" or "discarded" or "rejected" (depending on how you set up your .ctl file.

Code:

bad_records=`grep "not loaded" whatever1.log | awk '{print $NR}'`
echo "number of bad records=$bad_records"

xiamin · #7 01-16-2006

Hi Jim

Thank you very much..I appreciate your idea very much.

Regards
Hrishy

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
split files into specified number of output files	Migrainegirl	UNIX for Dummies Questions & Answers	4	05-13-2008 06:38 AM
Help please!Split files according to index	onthetopo	Shell Programming and Scripting	9	05-11-2007 06:37 AM
Split a file into 2 or more files	bobo	UNIX for Dummies Questions & Answers	4	01-16-2006 02:15 PM
awk command to split in to 2 files	m_subra_mani	Shell Programming and Scripting	3	12-13-2005 02:13 PM
killing parallel oracle process	oracle8	UNIX for Dummies Questions & Answers	1	10-09-2001 08:26 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode