Running scripts in parallel


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Running scripts in parallel
# 1  
Old 02-18-2008
Running scripts in parallel

Hi,


Iam having the scripts as follows. i jus want to run those in parallel.

main aim is to minimise the time for overall execution of the script.

now out.txt is having 1 lac records.

script1(split.sh)
Code:
split -1000 out.txt splitout
ls -A splitout* > filelist.txt

cat filelist.txt | while read fileln 
do
    parll.sh $fileln >> parll.nohup &
done

script2(parll.sh)
Code:
cat $1 | while read line
do
nohup grep -h $line filename.txt >> nohup.out       
done

main aim is to reduce the grep time from the filename.txt. so iam splitting the large file and passin gthe splitted file one by one and running the grep in parallel.


Still it is taking more than 3.5 hours for lac records itself. iam having more than 5 lac records.

so can anybody help me to reduce the time considerably.

Thanks.
nivas
# 2  
Old 02-18-2008
May I know what you want to do with out.txt?
The core motive behind this excersize?
Cheers!!!
# 3  
Old 02-19-2008
out.txt is a very large file to process

my motive is to quickly process the large file.

so iam using spliting to split the large file and small files have datas(file1) and iam using that data's and greping that line from file2 and writing to an output file.

To quick finish iam using nohup to run this spllited files paralley.

suggesion pls??
nivas
# 4  
Old 02-20-2008
It would be my pleasure to solve your problem, Smilie
However, just let me know the exact string, you want to search so that I can re-write it for faster processing.
I just want this in simple grep syntax i.e.: -
#grep "String" <filename>
# 5  
Old 02-20-2008
out.txt file looks like

QWERTYUI1234567890019289661
YUTTYYUI2763877779898588775
BVJHDFJFJ2678098802778897898


filename.txt will be

Code:
QWERTYUI1234567890019289661    hgsfd@williams.com                        287776666666778                        2008-878-398 -                                  62638678                                             37872fwweiiilslvcxmnmn


you can take the 2 scripts from my previous post.
nivas
# 6  
Old 02-20-2008
out.txt and filename.txt will have records in lacs.
nivas
# 7  
Old 02-21-2008
Nivas, piping cats will not yeild good performance though...!
google on 'piping cats', you will know why.

-ilan
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parallel bash scripts

Need some help to replace bash script with parallel to speed up job on multiple files (400files.list is the file contains the absolute path to those 400 files). The bash script is to run the same program over the files repetitively. My bash_script.sh is: for sample in `cat 400files.list`; do... (3 Replies)
Discussion started by: yifangt
3 Replies

2. Shell Programming and Scripting

Running script in Parallel

Hi Folks I have a doubt. I have a script which is running with 2 input parameters ./GetDSLnkCount.sh <jobnumber> Parmfile.txt I need to run the script in parallel for different jobnumbers. The commands are ./GetDSLnkCount.sh jnhuc14500 Parmfile.txt ./GetDSLnkCount.sh jnhuc14501... (3 Replies)
Discussion started by: morbid_angel
3 Replies

3. Shell Programming and Scripting

[Solved] Running scripts in parallel

i have script A and script B, both scripts have to run in parallel, my requirement is script A create table temp1, post creating it will run fr 4 hrs , script B has to start 0nly after creation of table temp1 ( which is done by script A) , again script B will run for 5 hrs if i run sequencially... (7 Replies)
Discussion started by: only4satish
7 Replies

4. Shell Programming and Scripting

[Solved] Running scripts in parallel that issue prompt

Hi all - I am totally stuck here :wall I have been asked to write a shell script that does a few little things and then reads from a config file and kicks off an instance of another script, say scriptB.ksh for each line in the config file. These should all be run in parallel. This is all fine but... (2 Replies)
Discussion started by: sjmolloy
2 Replies

5. UNIX for Dummies Questions & Answers

Running parallel process

i am having 4 process,have to run parallel and not after one by one. sample1.sh sample2.sh sample3.sh sample4.sh Thanks in advance. i (11 Replies)
Discussion started by: sagar_1986
11 Replies

6. Shell Programming and Scripting

Execute scripts in Parallel

Hi I want to execute few scripts in Parallel. There is a Master Script (MS.ksh) which will call internally all the scripts we need to run in Parallel. Say there are three set of scripts : ABC_1.ksh --> ABC_2.ksh --> ABC_3.ksh (execute ABC_2 when ABC_1 is successful ; Execute ABC_3 only when... (6 Replies)
Discussion started by: dashing201
6 Replies

7. Shell Programming and Scripting

Running jobs in parallel

I need to process 50 sqlplus scripts which are listed in a text file. I need to develop a shell script that'll read this file and run these sqlplus scripts. At any point of time, the number of sqlplus scripts running shouldn't exceed 6. If any of the sqlplus scripts completes successfully then... (17 Replies)
Discussion started by: gctex
17 Replies

8. Shell Programming and Scripting

Executing scripts in Parallel

Hi All, I have 3 shell scripts, Script1,Script2 and Script3. Now I want to run Script1 and Script2 in parallel and Script3 should depend on successful completion of both Script1 and Script2. Could you please suggest an approach of acheiving this... Thanks in advance (2 Replies)
Discussion started by: itsme_maverick
2 Replies

9. IP Networking

running servers parallel

I'm going to undertake a hardware refresh soon and I was wondering if it is possible to run two machines (X and Y) with the same hostname (but different IP addresses) on the same network? Server X is the original server and has an entry in DNS. Server Y is the new server and won't have an entry... (1 Reply)
Discussion started by: soliberus
1 Replies

10. Programming

running a parallel program

hi , i need to run a parallel program . for example; program1 { array=" the second program should called here : program 2" the execution should continue } the 2nd program should recieve an array of information as argument and it should... (4 Replies)
Discussion started by: bankpro
4 Replies
Login or Register to Ask a Question