Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to make this run in multiple threads

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to make this run in multiple threads
# 1  
Old 03-14-2011
Run Perl script in multiple threads
Hi,
I have a list of URLs in a csv file which I'm checking for page status. It just prints the URL and the status as output. This works perfectly fine.
I'm looking to run this in multiple threads to make this process faster.

I'm pretty new to Perl and I managed to complete this. It would be helpful if someone help me out with multiple threading.

Right now I use
Code:
perl checkURLs.pl check.csv > /tmp/results.txt

Code:
#!/usr/bin/perl

use strict;

while (my $line = <>){
  my $url = $line;
  $url =~ s/[ *\\]*\n//g;
  my $page = `curl -s -D - $url`;
  if ($page =~ /^HTTP\/\d.\d (\d+)/){
    print "$url status: $1\n";
  } else {
    print "Could not get page status\n";
  }
}

Last edited by kzenthil; 03-14-2011 at 10:22 PM..
# 2  
Old 03-15-2011
Don't PM me to get faster responses.

I don't know how to do threads in perl anyway.

You could split the list into sections and run a seperate instance of that script for each section of the list in the background, wait for them to finish, then combine the lists.
# 3  
Old 03-15-2011
Quote:
Originally Posted by Corona688
Don't PM me to get faster responses.
Ditto. (My perl is very rusty anyway.)
# 4  
Old 03-15-2011
you can read about Perl's threads using below:

on your cmd write:

Code:
perldoc threads

# 5  
Old 03-15-2011
Perl supports it's own implementation of Posix fork.
ie
Code:
while (<$urls_file>){
   chomp;
   my $child = fork();
   if ( $child ){                     # each child process will run for the value of $_ when it was called
      checkResource($_);  # assumes the curl script above is in a sub routine
      exit;                              # and exit when it has checked it.
   }
}

Last edited by Skrynesaver; 03-15-2011 at 07:45 AM.. Reason: added comments and made it stricture friendly
# 6  
Old 03-15-2011
Quote:
Originally Posted by Corona688
Don't PM me to get faster responses.

I don't know how to do threads in perl anyway.

You could split the list into sections and run a seperate instance of that script for each section of the list in the background, wait for them to finish, then combine the lists.
Thanks for your response. I apologize for the PM. I dint read the rules, my bad.

Quote:
Originally Posted by ahmad.diab
you can read about Perl's threads using below:

on your cmd write:

Code:
perldoc threads

Thanks for your response. I apologize for the PM.

Quote:
Originally Posted by Skrynesaver
Perl supports it's own implementation of Posix fork.
ie
Code:
while (<$urls_file>){
   chomp;
   my $child = fork();
   if ( $child ){                     # each child process will run for the value of $_ when it was called
      checkResource($_);  # assumes the curl script above is in a sub routine
      exit;                              # and exit when it has checked it.
   }
}

Thanks for your response. Will try this out and let you know.
# 7  
Old 03-15-2011
Quote:
Originally Posted by kzenthil
Thanks for your response. Will try this out and let you know.
One word of warning, you should perhaps get the checkResource function to write to a hash with the page name as key and set the value depending on response. Otherwise the unbuffered output could cross over itself as each will complete when it completes and the result could be difficult to read.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Run a shell script in a loop with max number of threads

hi guys. i have a question for you i have a one file and inside this file there are 1000 lines and each line is a linux command running this commands takes long time so i want to create one bash script and run this lines in a loop with max number of threads for example i want to run... (2 Replies)
Discussion started by: avtaritet
2 Replies

2. Shell Programming and Scripting

Need to create multiple threads

Hi , i need to run multiple scripts parallely ,on my server....i have 8 cpus . planning to run minimum of 6 scripts paralley ....could you please suggest someone . thanks in advance , (3 Replies)
Discussion started by: Huvan
3 Replies

3. Shell Programming and Scripting

How to start multiple threads in Solaris?

Hello, In a unix Solaris environment, (for simulation) how to start multiple threads (as Light Weight Process, not background process)? thanks, J. (7 Replies)
Discussion started by: seafan
7 Replies

4. Shell Programming and Scripting

Make script that run with argument if not run from configuration file argument

Hello, Is there any method thorugh which script can take argument if pass otherwise if argument doesn't pass then it takes the argument from the configuration file i.e I am workiing on a script which will run through crontab and the script will chekout the code ,zip and copy to the... (3 Replies)
Discussion started by: rohit22hamirpur
3 Replies

5. Shell Programming and Scripting

Multiple Threads/Tasks to run parallely using the shell script

Scenario: I have two PCs (named as A & B) which would send some traps to my third PC (named as C). In PC C, I have to write a shell script such that it should accept the datas from both the PC-A & B parallely. So my question is, is it possible to have two different child threads/tasks... (2 Replies)
Discussion started by: nthiruvenkatam
2 Replies

6. UNIX for Dummies Questions & Answers

Copy files in Multiple Threads

Hello all, I have a directory of files of varying sizes. I want to copy all these files in n number of threads to another directory such that each copy set is more or less the same size. Example : Say /mydirA It has around say 23 files of various sizes. Number of copy... (0 Replies)
Discussion started by: samoo
0 Replies

7. Programming

how can I get to know what threads run within process java.exe on windows

I am writing java application on windows. There are more than 100 threads run within java.exe. I want to know what threads run within process java.exe so that I can find out if there are abnormal java threads. (4 Replies)
Discussion started by: mika
4 Replies

8. IP Networking

How to choose Multiple process or Multiple threads?

Hi All, Please explain me when i have to use multiple process and when I have to use Multiple threads? Please give me an example.It will be very helpful for me. Thanks in advance. (0 Replies)
Discussion started by: ashleykumar
0 Replies

9. Programming

synchronizing multiple threads in unix

hi all! I wanted to know how to synchronize multiple threads in unix It would be better if someone give some code samples Thanx (1 Reply)
Discussion started by: bankpro
1 Replies

10. Programming

How to use pipe() in multiple threads?

Hi, I have a program that runs two threads in stead of two processes. I want to use pipe to redirect the output of the first thread to the input of the second thread. One thread is continuously writing to a pipe, and the other thread will read from the pipe. How do I do that? Is there... (2 Replies)
Discussion started by: wminghao
2 Replies
Login or Register to Ask a Question