How to make parallel processing rather than serial processing ??


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
How to make parallel processing rather than serial processing ??

Hello everybody,

I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel.

My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log files.

My problem is that I call my program in a bash script that tails my log files, like this :
Code:
#!/bin/bash
(tail -f /home/collectd/logs/file1 | perl /home/collectd/etc/collection.pl file1 arg2) &
(tail -f /home/collectd/logs/file2 | perl /home/collectd/etc/collection.pl file2 arg2) &
(tail -f /home/collectd/logs/file3 | perl /home/collectd/etc/collection.pl file3 arg2) &
(tail -f /home/collectd/logs/file4 | perl /home/collectd/etc/collection.pl file4 arg2) &
(tail -f /home/collectd/logs/file5 | perl /home/collectd/etc/collection.pl file5 arg2) &
while [1 -eq 1]; do
  sleep 50;
done

The problem is that it does not work very well. So I would like to make the "tail -f" parallel processing directly in my perl program.
Here is the perl program (collection.pl) I'd like to make parallel processing with :
Code:
#!/usr/bin/perl

use warnings;
use strict;

if ($#ARGV != 1){
    print "You did not give the two necessary arguments.\nUsage: perl my_prog.pl filename VOP\n";
    exit;
}

#name of the file
my $file_name=$ARGV[0];
#Summary type that I want to analyze
my $sum_type=$ARGV[1];

my $node_name;
my $operation_type;
my $operation_counter;

chomp($sum_type, $file_name);

my $previous_fh = select(STDOUT);
$| = 1;
select($previous_fh);

while (<STDIN>){
    chomp;

    #I use regular expressions here to retrieve the name of the server the type of operation and the counter value only if the summary type is VOP
    if (/\s$sum_type\sSUMMARY\s\[(\w+)\.\w+\]\s(\w+)\scnt\/(\d+)\s/) {
        $server_name=$1;
        $operation_type=$2;
        $operation_counter=$3;
        my $timestamp=time;

        #This is a print so that collectd (a program that I use to monitor my servers) can put a value in a RRD file from the print output
        print "PUTVAL $node_name/cvfs-$file_name/operations-$operation_type $timestamp:$operation_counter";
    }
}

I have seen that parallel processing could be done with fork or something like that but I don't really understand how to do it. Does anyone has an idea of how I could do that ?

Thanks by advance.

Last edited by Samb95; 08-16-2010 at 04:32 AM..
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Parallel processing
garethsays
I have 10,000 + files, each of which I need to zip using bzip2. Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished?... Shell Programming and Scripting
1
Shell Programming and Scripting
parallel processing
mad_man12
hi i am preparing a set of batches for a set of files sequentially There is a folder /xyz where all the files reside now all the files starting with 01 - will be appended for one below other to form a batch batch01 then all the files starting with 02 - will be appended for one below other to...... Shell Programming and Scripting
7
Shell Programming and Scripting
Need Help With Parallel Processing
imnewtothis23
Hi I am looking for some kind of feature in unix that will help me write a script that can invoke multiple processes in parallel. And make sure that the multiple parallel processes complete successfully before I proceed to the next step. Someone suggested something called timespid or...... Shell Programming and Scripting
6
Shell Programming and Scripting
parallel processing
audippa
Hi I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls Regards Audippa naidu.M... Shell Programming and Scripting
3
Shell Programming and Scripting
How to do parallel processing??
zing
Hi All, I am working on solaris 8 sparc machine with 2 cpu. I am trying to run my application which generates files. I run multiple instance of the application, but the results don't seem to show as if it were runing parallely. When i run the application once it takes 12 secs to generate a...... UNIX for Dummies Questions & Answers
1
UNIX for Dummies Questions & Answers