learn unix and linux commands

fastergrace · #1 05-07-2010 - Original Discussion by fastergrace

I'm trying to write a script that will parse the output of the iostat command in real time and place the output in csv file(s). I do have a programming background, but am relatively new to shell so I'm having difficulties determining how to proceed.

The cpu stats will go into one output file, and the device stats into another. I've a mind to take the output line by line, parse the string, format it, and write the output to its respective csv file. Maybe I could use if/else or a case statement to determine what to do with the string. I think it would be a significant bit less complicated if I didn't have to do this in real time.

Really, I'm just hoping someone can point me to some good resources to guide me to get this done. It's a little more complicated than the basic examples I've found so far. Any and all help is appreciated.

Franklin52 · #2 05-07-2010 - Original Discussion by fastergrace

Post the output of the iostat command and the desired output in the csv files.

Regards

fastergrace · #3 05-07-2010 - Original Discussion by fastergrace

Quote:

Originally Posted by Franklin52

Post the output of the iostat command and the desired output in the csv files.

Sure.

iostat output is below. This particular example is in five second intervals but the script will take in a variable that allows the user to specify whatever time interval in seconds they prefer.

Code:

-bash-3.00$ iostat 5 -x -t
Linux 2.6.9-42.0.3.ELsmp (wombat64)     07/05/10        _x86_64_        (2 CPU)

07/05/10 21:34:16
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          13.82    0.00    2.16    0.72    0.00   83.31

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
hda               0.00     3.89    0.11    0.34     5.33    33.92    87.03     0.28  629.52   2.97   0.13
hdb               0.16    20.18    1.14    4.13    81.66   194.55    52.38     1.24  234.53   2.00   1.05

07/05/10 21:34:21
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.40    0.00    0.60    0.10    0.00   98.90

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
hda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
hdb               0.00     2.99    0.00    0.80     0.00    30.34    38.00     0.00    1.25   1.25   0.10

07/05/10 21:34:26
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.50    0.00    0.30    0.20    0.00   99.00

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
hda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
hdb               0.00     3.60    0.20    0.60     1.60    33.60    44.00     0.00    5.25   5.25   0.42

pseudocoder · #4 05-08-2010 - Original Discussion by fastergrace

If you can accept small delay, e.g. 60 sec, then you might find following code useful.
You'll need to redirect the output of iostat to iostat.dat
I'd be glad if you would at least try it once and report about your experience.
Besides I have to say that my iostat version is completely different than yours,
but I did some manual simulation and it worked like a charm.
This is the reason why I chose 60 sec.
Maybe it will work for you in real time, if you set sleep time to 5 sec.?

Code:

#!/usr/bin/perl

my $data = 'iostat.dat' ;
my $cpucsv = 'cpu.csv' ;
my $devcsv = 'dev.csv' ;
my $count = 0 ;
my $date = '' ;
my $lastdate = '' ;
my @cleanarr = () ;
my @newcleanarr = () ;
my $lastarrsize = 0 ;

open(CPU,">$cpucsv") or die "Error: $!\n" ;
print CPU "date/time,%user,%nice,%system,%iowait,%steal,%idle\n" ;
close(CPU) ;

open(DEV,">$devcsv") or die "Error: $!\n" ;
print DEV "date/time,rrqm/s,wrqm/s,r/s,w/s,rsec/s,wsec/s,avgrq-sz,avgqu-sz,await,svctm,%util\n" ;
close(DEV) ;

while (1) {

print "Working...\n" ;

open(DATA,$data) or die "Error: $!\n" ;
my @dataarr = <DATA> ;
close(DATA);

my $arrsize = @dataarr ;

 if ($lastdate eq "") {

  $lastarrsize = $arrsize ;

  foreach (@dataarr) {
  s/^Linux.*//g ;
  s/^avg.*//g ;
  s/^Device.*//g ;
  s/^ *//g ;
  s/^\n$// ;
  s/  */ /g ;
  s/ /,/g ;
  push(@cleanarr,$_);}

  @newcleanarr = grep length( $_ ), @cleanarr ;

open(CPU,">>$cpucsv") or die "Error: $!\n" ;
open(DEV,">>$devcsv") or die "Error: $!\n" ;

  foreach (@newcleanarr) {
  chomp $_ ;

  $count++ ;

    if ($count eq 1) {
    s/,/ /g ;
    $date = $_ ;
    print CPU "$date," ; }

      if ($count eq 2) {
      print CPU "$_\n" ;   }

        if ($count eq 3) {
        print DEV "$date,$_\n" ; }

          if ($count eq 4) {
	  print DEV "$date,$_\n" ;
          $count=0;
          $lastdate = $date ; 
          $date='' ;       }

  }

close(CPU) ;
close(DEV) ;

 } else { 

 delete @dataarr[0..$lastarrsize-1] ;
 $lastarrsize = $arrsize ;

  @cleanarr = () ;
  @newcleanarr = () ;

  foreach (@dataarr) {
 
# s/^Linux.*//g ;
  s/^avg.*//g ;
  s/^Device.*//g ;
  s/^ *//g ;
  s/^\n$// ;
  s/  */ /g ;
  s/ /,/g ;
  push(@cleanarr,$_);}

  @newcleanarr = grep length( $_ ), @cleanarr ;

open(CPU,">>$cpucsv") or die "Error: $!\n" ;
open(DEV,">>$devcsv") or die "Error: $!\n" ;

  foreach (@newcleanarr) {
  chomp $_ ;

  $count++ ;

    if ($count eq 1) {
    s/,/ /g ;
    $date = $_ ;
    print CPU "$date," ; }

      if ($count eq 2) {
      print CPU "$_\n" ;   }

        if ($count eq 3) {
        print DEV "$date,$_\n" ; }

          if ($count eq 4) {
          print DEV "$date,$_\n" ;
          $count=0;
          $lastdate = $date ;
          $date='' ;       }

  }

close(CPU) ;
close(DEV) ;

 }

print "Sleeping 60 sec...\n";
sleep(60);

}

Franklin52 · #5 05-08-2010 - Original Discussion by fastergrace

With awk:

Code:

iostat 5 -x -t |
awk '
BEGIN {
  print "date/time,%user,%nice,%system,%iowait,%steal,%idle" > "cpu_stats"
  print "date/time,rrqm/s,wrqm/s,r/s,w/s,rsec/s,wsec/s,avgrq-sz,avgqu-sz ,await,svctm,%util" > "device_stats"
}
NF==2{
  s=$0
  getline; getline; $1=$1
  print s "," $0  > "cpu_stats"
}
/Device:/{
  getline; $1=$1
  print s "," $0 > "device_stats"
  getline; $1=$1
  print s "," $0 > "device_stats"
}'

fastergrace · #6 05-10-2010 - Original Discussion by fastergrace

Thanks so much for the guidance, guys. I will give both a shot and let you know how it goes. Ideally, I want the data to be parsed and written in real time without a buffer, although I did consider going that route.

fastergrace · #7 05-13-2010 - Original Discussion by fastergrace

So, this is the final product:

Code:

lineCount=$(iostat | wc -l)
numDevices=$(expr $lineCount - 7);

iostat $interval -x -t |
awk -v awkCpuFile=$cpuFile -v awkDeviceFile=$deviceFile -v awkNumDevices=$numDevices '
BEGIN {
	print "date/time,%user,%nice,%system,%iowait,%steal,%idle" > awkCpuFile
	print "date/time,device,rrqm/s,wrqm/s,r/s,w/s,rsec/s,wsec/s,avgrq-sz,avgqu-sz,await,svctm,%util" > awkDeviceFile
}
NF==2{
	s=$0
	getline; getline; $1=$1
	gsub(/ /,",",$0)
	print s "," $0 > awkCpuFile	
}
/Device:/{
	for (i=0; i<awkNumDevices; i++) {
		getline; $1=$1
		gsub(/ /,",",$0)
		print s "," $0 > awkDeviceFile
	}
}'

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
problem with real-time	quant	Programming	0	09-12-2009 10:48 PM
Real time problems	kurva	Solaris	2	04-13-2009 09:45 AM

learn unix and linux commands

Parsing iostat in real time

Shell Programming and Scripting

The Following User Says Thank You to pseudocoder For This Useful Post:
fastergrace (05-13-2010)

The Following User Says Thank You to Franklin52 For This Useful Post:
fastergrace (05-13-2010)