Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Parsing iostat in real time

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parsing iostat in real time
# 1  
Old 05-07-2010
Question Parsing iostat in real time
I'm trying to write a script that will parse the output of the iostat command in real time and place the output in csv file(s). I do have a programming background, but am relatively new to shell so I'm having difficulties determining how to proceed.

The cpu stats will go into one output file, and the device stats into another. I've a mind to take the output line by line, parse the string, format it, and write the output to its respective csv file. Maybe I could use if/else or a case statement to determine what to do with the string. I think it would be a significant bit less complicated if I didn't have to do this in real time.

Really, I'm just hoping someone can point me to some good resources to guide me to get this done. It's a little more complicated than the basic examples I've found so far. Any and all help is appreciated.
# 2  
Old 05-07-2010
Post the output of the iostat command and the desired output in the csv files.

Regards
# 3  
Old 05-07-2010
Quote:
Originally Posted by Franklin52
Post the output of the iostat command and the desired output in the csv files.
Sure.

iostat output is below. This particular example is in five second intervals but the script will take in a variable that allows the user to specify whatever time interval in seconds they prefer.
Code:
-bash-3.00$ iostat 5 -x -t
Linux 2.6.9-42.0.3.ELsmp (wombat64)     07/05/10        _x86_64_        (2 CPU)

07/05/10 21:34:16
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          13.82    0.00    2.16    0.72    0.00   83.31

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
hda               0.00     3.89    0.11    0.34     5.33    33.92    87.03     0.28  629.52   2.97   0.13
hdb               0.16    20.18    1.14    4.13    81.66   194.55    52.38     1.24  234.53   2.00   1.05

07/05/10 21:34:21
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.40    0.00    0.60    0.10    0.00   98.90

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
hda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
hdb               0.00     2.99    0.00    0.80     0.00    30.34    38.00     0.00    1.25   1.25   0.10

07/05/10 21:34:26
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.50    0.00    0.30    0.20    0.00   99.00

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
hda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
hdb               0.00     3.60    0.20    0.60     1.60    33.60    44.00     0.00    5.25   5.25   0.42

This is what I'd like the cpu statistics to look like:
Code:
date/time,%user,%nice,%system,%iowait,%steal,%idle
07/05/10 21:34:16,13.82,0.00,2.16,0.72,0.00,83.31
07/05/10 21:34:21,0.40,0.00,0.60,0.10,0.00,98.90
07/05/10 21:34:26,0.50,0.00,0.30,0.20,0.00,99.00

And the device stats:
Code:
date/time,rrqm/s,wrqm/s,r/s,w/s,rsec/s,wsec/s,avgrq-sz,avgqu-sz ,await,svctm,%util
07/05/10 21:34:16,hda,0.00,3.89,0.11,0.34,5.33,33.92,87.03,0.28,629.52,2.97,0.13
07/05/10 21:34:16,hdb,0.16,20.18,1.14,4.13,81.66,194.55,52.38,1.24,234.53,2.00,1.05
07/05/10 21:34:21,hda,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
07/05/10 21:34:21,hdb,0.00,2.99,0.00,0.80,0.00,30.34,38.00,0.00,1.25, 1.25,0.10
07/05/10 21:34:26,hda,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
07/05/10 21:34:26,hdb,0.00,3.60,0.20,0.60,1.60,33.60,44.00,0.00,5.25,5.25,0.42



I can probably figure out how to use sed or awk to parse the strings, honestly my confusion stems more from the logistics of how to take the output and manipulate it in rt. I can run two separate instances of iostat, one which pulls cpu stats (iostat -c) and one which pulls device stats (iostat -d), but it is preferable to parse the output from one.
# 4  
Old 05-08-2010
If you can accept small delay, e.g. 60 sec, then you might find following code useful.
You'll need to redirect the output of iostat to iostat.dat
I'd be glad if you would at least try it once and report about your experience.
Besides I have to say that my iostat version is completely different than yours,
but I did some manual simulation and it worked like a charm.
This is the reason why I chose 60 sec.
Maybe it will work for you in real time, if you set sleep time to 5 sec.?
Code:
#!/usr/bin/perl

my $data = 'iostat.dat' ;
my $cpucsv = 'cpu.csv' ;
my $devcsv = 'dev.csv' ;
my $count = 0 ;
my $date = '' ;
my $lastdate = '' ;
my @cleanarr = () ;
my @newcleanarr = () ;
my $lastarrsize = 0 ;

open(CPU,">$cpucsv") or die "Error: $!\n" ;
print CPU "date/time,%user,%nice,%system,%iowait,%steal,%idle\n" ;
close(CPU) ;

open(DEV,">$devcsv") or die "Error: $!\n" ;
print DEV "date/time,rrqm/s,wrqm/s,r/s,w/s,rsec/s,wsec/s,avgrq-sz,avgqu-sz,await,svctm,%util\n" ;
close(DEV) ;

while (1) {

print "Working...\n" ;

open(DATA,$data) or die "Error: $!\n" ;
my @dataarr = <DATA> ;
close(DATA);

my $arrsize = @dataarr ;

 if ($lastdate eq "") {

  $lastarrsize = $arrsize ;

  foreach (@dataarr) {
  s/^Linux.*//g ;
  s/^avg.*//g ;
  s/^Device.*//g ;
  s/^ *//g ;
  s/^\n$// ;
  s/  */ /g ;
  s/ /,/g ;
  push(@cleanarr,$_);}

  @newcleanarr = grep length( $_ ), @cleanarr ;

open(CPU,">>$cpucsv") or die "Error: $!\n" ;
open(DEV,">>$devcsv") or die "Error: $!\n" ;

  foreach (@newcleanarr) {
  chomp $_ ;

  $count++ ;

    if ($count eq 1) {
    s/,/ /g ;
    $date = $_ ;
    print CPU "$date," ; }

      if ($count eq 2) {
      print CPU "$_\n" ;   }

        if ($count eq 3) {
        print DEV "$date,$_\n" ; }

          if ($count eq 4) {
	  print DEV "$date,$_\n" ;
          $count=0;
          $lastdate = $date ; 
          $date='' ;       }

  }

close(CPU) ;
close(DEV) ;

 } else { 

 delete @dataarr[0..$lastarrsize-1] ;
 $lastarrsize = $arrsize ;

  @cleanarr = () ;
  @newcleanarr = () ;

  foreach (@dataarr) {
 
# s/^Linux.*//g ;
  s/^avg.*//g ;
  s/^Device.*//g ;
  s/^ *//g ;
  s/^\n$// ;
  s/  */ /g ;
  s/ /,/g ;
  push(@cleanarr,$_);}

  @newcleanarr = grep length( $_ ), @cleanarr ;

open(CPU,">>$cpucsv") or die "Error: $!\n" ;
open(DEV,">>$devcsv") or die "Error: $!\n" ;

  foreach (@newcleanarr) {
  chomp $_ ;

  $count++ ;

    if ($count eq 1) {
    s/,/ /g ;
    $date = $_ ;
    print CPU "$date," ; }

      if ($count eq 2) {
      print CPU "$_\n" ;   }

        if ($count eq 3) {
        print DEV "$date,$_\n" ; }

          if ($count eq 4) {
          print DEV "$date,$_\n" ;
          $count=0;
          $lastdate = $date ;
          $date='' ;       }

  }

close(CPU) ;
close(DEV) ;

 }

print "Sleeping 60 sec...\n";
sleep(60);

}

This User Gave Thanks to pseudocoder For This Post:
fastergrace
# 5  
Old 05-08-2010
With awk:
Code:
iostat 5 -x -t |
awk '
BEGIN {
  print "date/time,%user,%nice,%system,%iowait,%steal,%idle" > "cpu_stats"
  print "date/time,rrqm/s,wrqm/s,r/s,w/s,rsec/s,wsec/s,avgrq-sz,avgqu-sz ,await,svctm,%util" > "device_stats"
}
NF==2{
  s=$0
  getline; getline; $1=$1
  print s "," $0  > "cpu_stats"
}
/Device:/{
  getline; $1=$1
  print s "," $0 > "device_stats"
  getline; $1=$1
  print s "," $0 > "device_stats"
}'

This User Gave Thanks to Franklin52 For This Post:
fastergrace
# 6  
Old 05-10-2010
Thanks so much for the guidance, guys. I will give both a shot and let you know how it goes. Ideally, I want the data to be parsed and written in real time without a buffer, although I did consider going that route.
# 7  
Old 05-13-2010
So, this is the final product:
Code:
lineCount=$(iostat | wc -l)
numDevices=$(expr $lineCount - 7);

iostat $interval -x -t |
awk -v awkCpuFile=$cpuFile -v awkDeviceFile=$deviceFile -v awkNumDevices=$numDevices '
BEGIN {
	print "date/time,%user,%nice,%system,%iowait,%steal,%idle" > awkCpuFile
	print "date/time,device,rrqm/s,wrqm/s,r/s,w/s,rsec/s,wsec/s,avgrq-sz,avgqu-sz,await,svctm,%util" > awkDeviceFile
}
NF==2{
	s=$0
	getline; getline; $1=$1
	gsub(/ /,",",$0)
	print s "," $0 > awkCpuFile	
}
/Device:/{
	for (i=0; i<awkNumDevices; i++) {
		getline; $1=$1
		gsub(/ /,",",$0)
		print s "," $0 > awkDeviceFile
	}
}'

I made it a little more dynamic. This was a great stepping stone for me to learn more about shell scripting, and more specifically awk. Thanks again for the help!
Login or Register to Ask a Question

Previous Thread | Next Thread

6 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Converting real time to epoch time

# date +%s -d "Mon Feb 11 02:26:04" 1360567564 # perl -e 'print scalar localtime(1360567564), "\n";' Mon Feb 11 02:26:04 2013 the epoch conversion is working fine. but one of my application needs 13 digit epoch time as input 1359453135154 rather than 10 digit epoch time 1360567564... (3 Replies)
Discussion started by: vivek d r
3 Replies

2. Shell Programming and Scripting

Parsing iostat data

Hello, I searched the forum and found the parsing iostat data in real time but that does not help me. What I would like to do is to parse the iostat data that has been gathered. Below is an example of the iostat output: System configuration: lcpu=4 drives=5 ent=0.10 paths=20 vdisks=2 ... (11 Replies)
Discussion started by: tommyd
11 Replies

3. UNIX for Dummies Questions & Answers

Real time processing

Hi Not sure if this can be achieved by unix , but still would like to know if there is any way by which I can do the below given logic cat sam1 > out1 cat sam2 > out2 when either one of this finished the the next file shd be written in that file, meaning cat sam3 >> out1/out2... (2 Replies)
Discussion started by: Sri3001
2 Replies

4. Shell Programming and Scripting

Shell script to convert epoch time to real time

Dear experts, I have an epoch time input file such as : - 1302451209564 1302483698948 1302485231072 1302490805383 1302519244700 1302492787481 1302505299145 1302506557022 1302532112140 1302501033105 1302511536485 1302512669550 I need the epoch time above to be converted into real... (4 Replies)
Discussion started by: aismann
4 Replies

5. Solaris

Real time problems

Hi friends, I am new to solaris and looking for a job, when ever i attend interview i get most of the questions on real time problems, every one sak me the same questions what are the problems you face daily.. and what are the types? i know few like, disk extension,swap memory increasing,... (2 Replies)
Discussion started by: kurva
2 Replies

6. UNIX for Advanced & Expert Users

EPOCH to real time?

hi all :confused: i am wondering if there is a way to convert from EPOCH time to the standard tim, may be using a script or some thing else??????? thanks............................ (5 Replies)
Discussion started by: TheEngineer
5 Replies
Login or Register to Ask a Question