How to make parallel processing rather than serial processing ??

Tags
shell scripts

 
Thread Tools Search this Thread
# 1  
Old 08-16-2010
How to make parallel processing rather than serial processing ??

Hello everybody,

I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel.

My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log files.

My problem is that I call my program in a bash script that tails my log files, like this :
Code:
#!/bin/bash
(tail -f /home/collectd/logs/file1 | perl /home/collectd/etc/collection.pl file1 arg2) &
(tail -f /home/collectd/logs/file2 | perl /home/collectd/etc/collection.pl file2 arg2) &
(tail -f /home/collectd/logs/file3 | perl /home/collectd/etc/collection.pl file3 arg2) &
(tail -f /home/collectd/logs/file4 | perl /home/collectd/etc/collection.pl file4 arg2) &
(tail -f /home/collectd/logs/file5 | perl /home/collectd/etc/collection.pl file5 arg2) &
while [1 -eq 1]; do
  sleep 50;
done

The problem is that it does not work very well. So I would like to make the "tail -f" parallel processing directly in my perl program.
Here is the perl program (collection.pl) I'd like to make parallel processing with :
Code:
#!/usr/bin/perl

use warnings;
use strict;

if ($#ARGV != 1){
    print "You did not give the two necessary arguments.\nUsage: perl my_prog.pl filename VOP\n";
    exit;
}

#name of the file
my $file_name=$ARGV[0];
#Summary type that I want to analyze
my $sum_type=$ARGV[1];

my $node_name;
my $operation_type;
my $operation_counter;

chomp($sum_type, $file_name);

my $previous_fh = select(STDOUT);
$| = 1;
select($previous_fh);

while (<STDIN>){
    chomp;

    #I use regular expressions here to retrieve the name of the server the type of operation and the counter value only if the summary type is VOP
    if (/\s$sum_type\sSUMMARY\s\[(\w+)\.\w+\]\s(\w+)\scnt\/(\d+)\s/) {
        $server_name=$1;
        $operation_type=$2;
        $operation_counter=$3;
        my $timestamp=time;

        #This is a print so that collectd (a program that I use to monitor my servers) can put a value in a RRD file from the print output
        print "PUTVAL $node_name/cvfs-$file_name/operations-$operation_type $timestamp:$operation_counter";
    }
}

I have seen that parallel processing could be done with fork or something like that but I don't really understand how to do it. Does anyone has an idea of how I could do that ?

Thanks by advance.

Last edited by Samb95; 08-16-2010 at 05:32 AM..

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Parallel processing garethsays Shell Programming and Scripting 1 03-19-2016 09:02 AM
Parallel processing and error checking wahi80 Shell Programming and Scripting 7 02-16-2016 09:31 PM
Parallel processing for functions in xargs jawsnnn Shell Programming and Scripting 9 08-07-2015 04:56 PM
Parallel processing - continued kumarjt Shell Programming and Scripting 3 08-21-2013 06:10 AM
Implement parallel processing kumarjt Shell Programming and Scripting 13 07-20-2013 06:42 AM
Parallel processing pthread temursalin Programming 0 07-12-2012 04:44 PM
Parallel processing in bash? jamie_123 Shell Programming and Scripting 5 06-14-2012 12:31 PM
PARALLEL PROCESSING IN PERL gvk25 Shell Programming and Scripting 0 04-14-2012 02:01 AM
Algorithms for Parallel Processing azar.zorn Programming 2 07-03-2011 12:18 PM
script parallel processing machpee Shell Programming and Scripting 2 06-04-2011 11:04 PM
parallel processing mad_man12 Shell Programming and Scripting 7 09-07-2009 03:56 AM
Need Help With Parallel Processing imnewtothis23 Shell Programming and Scripting 6 08-07-2008 02:15 AM
how to achieve following parallel processing thru unix manas_ranjan Shell Programming and Scripting 3 09-04-2007 08:54 AM
parallel processing audippa Shell Programming and Scripting 3 03-01-2007 12:01 PM
How to do parallel processing?? zing UNIX for Dummies Questions & Answers 1 06-23-2003 01:47 PM