Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

split process files in parallel and rejoin

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting split process files in parallel and rejoin
# 1  
Old 01-10-2006
split process files in parallel and rejoin
Hi

I need to split a huge file into multiple smaller files using split command.

After that i need to process each file in the back ground with sql loader .Sql loader is a utlity to load CSV files into oracle .

Check the status of each of these sqlloaders and then after sucessfull completion join the file back again.

Any idea on how to do run the process in background and then check for their sucessful execution

regards
Hrishy
# 2  
Old 01-10-2006
Script :
======

#!/usr/bin/ksh

split -l 1000 hugefile.txt -a hugefile

for fname in hugefile*
do
sqlldr user/password control=xyz.ctl log=${fname}.log data=$fname &
done

echo "Waiting for background process to complete...."

wait

echo "All files loaded..!!"


For finding the status, you can simply grep all the log files or count the lines in bad file.... not sure whether there is a way to trap directly from sqlldr command..

The above split will not actually split the original file, hence no need to join the splitted files back... simply delete the splitted files.

I have not tested the above stuff, correct if you have any syntax errors..
# 3  
Old 01-11-2006
Hi Mahendar

You almost compeletd the assignement for me :-).Thank you very much.

What i am still wundering is that how do i check each background process is there any away say something like

job -l and then look for done command if it fails then it will have something else for the return code.How di build this into the script.

regards
Hrishy
# 4  
Old 01-11-2006
Quote:
Originally Posted by xiamin
You almost compeletd the assignement for me :-).Thank you very much.
School/University assignment ?

If it is take a look at point 6 of the rules

(6) Do not post classroom or homework problems.
# 5  
Old 01-11-2006
Hi Vino

Nope its not a school assignement problem :-).I am sorry if my wordings sounded like that.

I knwo that we can submit the jobs in background using the & commnad but how do we monitor each job ?

something like job -l and then what how do i know if it sucesfully completed or not if it failed which is the one that failed.

regards
Hrishy
# 6  
Old 01-11-2006
the {$fname}.log files will each have a section like this at the end of the file:
Code:
Table UABBDBT:
  1 Row successfully loaded.
  0 Rows not loaded due to data errors.
  0 Rows not loaded because all WHEN clauses were failed.
  0 Rows not loaded because all fields were null.


Space allocated for bind array:                 255420 bytes(55 rows)
Read   buffer bytes: 1048576

Total logical records skipped:          0
Total logical records read:             1
Total logical records rejected:         0
Total logical records discarded:        0

Run began on Tue Jul 19 11:28:38 2005
Run ended on Tue Jul 19 11:28:39 2005

Elapsed time was:     00:00:00.45
CPU time was:         00:00:00.14

grep for "not loaded" or "discarded" or "rejected" (depending on how you set up your .ctl file.
Code:
bad_records=`grep "not loaded" whatever1.log | awk '{print $NR}'`
echo "number of bad records=$bad_records"

# 7  
Old 01-16-2006
Hi Jim

Thank you very much..I appreciate your idea very much.

Regards
Hrishy
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Automate splitting of files , scp files as each split completes and combine files on target server

i use the split command to split a one terabyte backup file into 10 chunks of 100 GB each. The files are split one after the other. While the files is being split, I will like to scp the files one after the other as soon as the previous one completes, from server A to Server B. Then on server B ,... (2 Replies)
Discussion started by: malaika
2 Replies

2. UNIX for Beginners Questions & Answers

Split and Rename Split Files

Hello, I need to split a file by number of records and rename each split file with actual filename pre-pended with 3 digit split number. What I have tried is the below command with 2 digit numeric value split -l 3 -d abc.txt F (# Will Produce split Files as F00 F01 F02) How to produce... (19 Replies)
Discussion started by: techedipro
19 Replies

3. Shell Programming and Scripting

Monitoring processes in parallel and process log file after process exits

I am writing a script to kick off a process to gather logs on multiple nodes in parallel using "&". These processes create individual log files. Which I would like to filter and convert in CSV format after they are complete. I am facing following issues: 1. Monitor all Processes parallelly.... (5 Replies)
Discussion started by: shunya
5 Replies

4. Shell Programming and Scripting

Process split files with same name by group

File examples f17_mar_01_02_03_04_fsw1.xml f17_mar_01_02_03_04_fsw2.xml f17_mar_01_02_03_04_fsw3.xml f17_feb_13_20_49_06_fsw1.xml f17_feb_13_20_49_06_fsw2.xml f17_feb_13_20_49_06_fsw3.xml I have many xml files that are grouped with same file name, but are numbered from 1-to-many... (2 Replies)
Discussion started by: aachave1
2 Replies

5. Shell Programming and Scripting

Make process parallel

Hi, I have a file which has a list of 200 tables e.g: table.txt I need to do a count for each table and store it in a file. So I did something like this: for TABLE in `cat table.txt` do T_CNT=$(sqlplus -s -l / as sysdba <<EOF set echo off heading off feadback off SELECT count(*) FROM... (1 Reply)
Discussion started by: wahi80
1 Replies

6. UNIX for Dummies Questions & Answers

Running parallel process

i am having 4 process,have to run parallel and not after one by one. sample1.sh sample2.sh sample3.sh sample4.sh Thanks in advance. i (11 Replies)
Discussion started by: sagar_1986
11 Replies

7. Programming

Parallel process in java

Hello; Please Are both threads execute in parallel? Thank you (4 Replies)
Discussion started by: chercheur857
4 Replies

8. Shell Programming and Scripting

Parallel process in Perl

HI All, I have scenerio where I need to call sub modules through for loop for (i=0; i<8000 ;i++) { .. BLOCKA } BLOCKA { .. .. subroutine 1; subroutine 2; } I want this to be run in parallel process1 BLOCKA { (6 Replies)
Discussion started by: gvk25
6 Replies

9. UNIX and Linux Applications

How can i see if a unix Process Aplication i.e oracle is running in parallel

There is a unix process process in oracle running and i see running by typing ps -fea|grep GE_CLIENTES. The question is How can i see if this process is running in paralel. I dont know with a Unix command or specifically its a comand from Oracle. I kow a Parallel process ia a process that... (1 Reply)
Discussion started by: alexcol
1 Replies

10. UNIX for Dummies Questions & Answers

killing parallel oracle process

Hi guys: I have a an oracle job which uses 10 parallel hints and would like to killit when it hangs. I want to kill all the processes that have been spawned. what I do right now is get the pid of the scheduler process which initiated theis job and the do a ps -ef| grep 'pid' and trace through... (1 Reply)
Discussion started by: oracle8
1 Replies
Login or Register to Ask a Question