The UNIX and Linux Forums  


Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
.
google unix.com



Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Executing scripts in Parallel itsme_maverick Shell Programming and Scripting 2 06-06-2008 12:23 PM
running servers parallel soliberus IP Networking 1 06-03-2008 08:45 AM
Running scripts parallely santho UNIX for Dummies Questions & Answers 4 07-23-2006 01:42 PM
Running three scripts parallelly anwarsait Shell Programming and Scripting 1 07-18-2006 02:20 AM
running a parallel program bankpro High Level Programming 4 01-20-2006 08:20 PM

Closed Thread
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Bulgarian Greek Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 02-18-2008
nivas
Guest
  
 

Posts: n/a
Bits: 0 [Banking]
Running scripts in parallel

Hi,


Iam having the scripts as follows. i jus want to run those in parallel.

main aim is to minimise the time for overall execution of the script.

now out.txt is having 1 lac records.

script1(split.sh)

Code:
split -1000 out.txt splitout
ls -A splitout* > filelist.txt

cat filelist.txt | while read fileln 
do
    parll.sh $fileln >> parll.nohup &
done

script2(parll.sh)

Code:
cat $1 | while read line
do
nohup grep -h $line filename.txt >> nohup.out       
done

main aim is to reduce the grep time from the filename.txt. so iam splitting the large file and passin gthe splitted file one by one and running the grep in parallel.


Still it is taking more than 3.5 hours for lac records itself. iam having more than 5 lac records.

so can anybody help me to reduce the time considerably.

Thanks.
  #2 (permalink)  
Old 02-18-2008
Thunderbolt Thunderbolt is offline
Registered User
  
 

Join Date: Sep 2006
Location: New Delhi - India
Posts: 5
May I know what you want to do with out.txt?
The core motive behind this excersize?
Cheers!!!
  #3 (permalink)  
Old 02-19-2008
nivas
Guest
  
 

Posts: n/a
Bits: 0 [Banking]
out.txt is a very large file to process

my motive is to quickly process the large file.

so iam using spliting to split the large file and small files have datas(file1) and iam using that data's and greping that line from file2 and writing to an output file.

To quick finish iam using nohup to run this spllited files paralley.

suggesion pls??
  #4 (permalink)  
Old 02-20-2008
Thunderbolt Thunderbolt is offline
Registered User
  
 

Join Date: Sep 2006
Location: New Delhi - India
Posts: 5
It would be my pleasure to solve your problem,
However, just let me know the exact string, you want to search so that I can re-write it for faster processing.
I just want this in simple grep syntax i.e.: -
#grep "String" <filename>
  #5 (permalink)  
Old 02-20-2008
nivas
Guest
  
 

Posts: n/a
Bits: 0 [Banking]
out.txt file looks like

QWERTYUI1234567890019289661
YUTTYYUI2763877779898588775
BVJHDFJFJ2678098802778897898


filename.txt will be


Code:
QWERTYUI1234567890019289661    hgsfd@williams.com                        287776666666778                        2008-878-398 -                                  62638678                                             37872fwweiiilslvcxmnmn


you can take the 2 scripts from my previous post.
  #6 (permalink)  
Old 02-20-2008
nivas
Guest
  
 

Posts: n/a
Bits: 0 [Banking]
out.txt and filename.txt will have records in lacs.
Closed Thread

Bookmarks

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On




All times are GMT -4. The time now is 09:07 AM.


Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0