Graphing data with awk/while loop


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Graphing data with awk/while loop
# 1  
Old 11-16-2013
Graphing data with awk/while loop

I need to graph data that's in a file. however, i only know of a way to graph data as it comes (meaning, as it is spat out by whatever process or job). i dont know how to graph data in a file.

im using rrdtool to graph data. and i was wondering if anyone know of a way to pass a file (the datafile), to the rrdtool so it graphs the necessary information.

the datafile contains several lines that look like this:

Code:
1384049064,64,83,0

The first field is the epoch time, the timestamp. the numbers that needs to be graphed are the numbers in the second and third column.

the second column number is the pl, the third column number is the rtt. I can think of a while loop to use on this, but considering i intend to run this on a large datafile, i foresee that being inefficient.

below is a snippet of what i'm using:

Code:
/usr/bin/rrdtool graph ${spdir}/datafile_hour.png \
-w 785 -h 120 -a PNG \
--slope-mode \
--start -1h \
--title=Great \
--vertical-label=UserLoginAttempts \
--lower=0 \
DEF:pl=latencydb.rrd:pl:AVERAGE \
DEF:rtt=latencydb.rrd:rtt:AVERAGE \
CDEF:tsuccess=pl,300,* \
CDEF:tfailure=rtt,300,* \
AREA:tsuccess#00FF00:Successfulattempts \
STACK:tfailure#FF0000:Failedattempts \
COMMENT:"pkt loss1\:" \
GPRINT:tsuccess:AVERAGE:successfulattempts"%6.2lf" \
COMMENT:"pkt loss2\:" \
GPRINT:tfailure:AVERAGE:failureattempts"%6.2lf" \


so I was going to run my while loop to do something like this:
Code:
while read line
do
pl=$(echo $line | awk -F"," '{print $2}')
rtt=$(echo $line | awk -F"," '{print $3}')
.... then i would run the rrdtool command here with the values fed in line by line
....
done < datafile

# 2  
Old 11-16-2013
You'll be much more efficient if you do like
Code:
while IFS=, read datetime pl rtt rest
    do .... then i would run the rrdtool command here with the values fed in line by line ....
    done < datafile

, but - are you going to call rrdtool for every single data pair?
This User Gave Thanks to RudiC For This Post:
# 3  
Old 11-16-2013
Quote:
Originally Posted by RudiC
You'll be much more efficient if you do like
Code:
while IFS=, read datetime pl rtt rest do .... then i would run the rrdtool command here with the values fed in line by line .... done < datafile

, but - are you going to call rrdtool for every single data pair?
for every line in the datafile, yes. which is what scares me. the most i would have to go in the data file is no more than a month.

so let's assume there's about 80,000 lines in the data file. i would have to run the rrdtool tool 80,000 times. Smilie

i found this snippet php code, which does something similar, but the information it is graphing is random information. not real data. it was a snippet code used as an example. was hoping i can use the logic behind this code, apply it to awk, and get it to do what I want:

Code:
<?php
  // Fetch current time
  $now = time();

  // Simulate last 180 days of login, with a step of 5 minutes
  for ($t=$now - (3600 * 24 * 180); $t<=$now; $t+=300) {
    $success = rand(0, 10);
    $failure = rand(0, 5);
    $ret = rrd_update("login.rrd", "$t:$success:$failure");
  }

?>

# 4  
Old 11-17-2013
There is one more good tool called GMT, if you are debain user you can install using sudo sudo apt-get install gmt once it's installed set GMT path in your bashrc like this export PATH=$PATH:/usr/lib/gmt/bin

there after you can do xy plot, contour plot, even 3D using this tool, this link may be useful to you
http://gmt.soest.hawaii.edu/

Here I will give one small example of xy plot using gmt function psxy

Code:
#!/bin/bash

# -R  --> Range that is x axis 1 to 10 y axis -1 to 1
# -JX --> Projection you can change according to your need
# -B  --> Annotation spacing is 1 in x axis and 0.1 in y axis
# -X  --> X position currenty +5 
# -W4.5,blue --> 4.5 is line thickness and blue is line color
# test.ps -- > output file

# Generate sample file 
awk 'BEGIN{for(i=1;i<=10;i++)print i,sin(i)}' >xy.txt

# Plot
psxy xy.txt -R1/10/-1/1 -JX20/15 -Ba1:"X axis":/a0.1:"Y axis": -X5 -W4.5,blue >test.ps

# You can also use pipe like this
# awk 'BEGIN{for(i=1;i<=10;i++)print i,sin(i)}' | psxy -R1/10/-1/1 -JX20/15 -Ba1:"X axis":/a0.1:"Y axis": -X5 -W4.5,blue >test.ps

# Program to open output ps file you can use gv(Ghost Viewer) or okular also
evince test.ps

I am here attaching output ps file if you feel this will helpful go through it
This User Gave Thanks to Akshay Hegde For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need help in awk: running a loop with one column and segregate data 4 each uniq value in that field

Hi All, I have a file like this(having 2 column). Column 1: like a,b,c.... Column 2: having numbers. I want to segregate those numbers based on column 1. Example: file. a 5 b 9 b 620 a 710 b 230 a 330 b 1910 (4 Replies)
Discussion started by: Raza Ali
4 Replies

2. Shell Programming and Scripting

awk loop using array:wish to store array values from loop for use outside loop

Here's my code: awk -F '' 'NR==FNR { if (/time/ && $5>10) A=$2" "$3":"$4":"($5-01) else if (/time/ && $5<01) A=$2" "$3":"$4-01":"(59-$5) else if (/time/ && $5<=10) A=$2" "$3":"$4":0"($5-01) else if (/close/) { B=0 n1=n2; ... (2 Replies)
Discussion started by: klane
2 Replies

3. Shell Programming and Scripting

Running awk and while loop on data

so this is what i want to do. i have data that looks like this: DATA: Array: A Interface Type: SAS Unused Space: 0 MB Status: Failed Logical Drive: 2 Size: 279.4 GB Fault Tolerance: RAID 1 Heads: 255 Sectors Per Track: 32... (2 Replies)
Discussion started by: SkySmart
2 Replies

4. Infrastructure Monitoring

How to install Cacti (a system monitoring and graphing solution) with Nginx?

Here's a short tutorial on installing Cacti with Nginx on Linux. (0 Replies)
Discussion started by: notblog
0 Replies

5. Shell Programming and Scripting

Sum from successive lines following date header to create data for graphing connections

Hi, I have a log file created from a load balancer showing connections to each member of a two member pool with the following format (where first field is source IP, second field is load balanced IP address and third field is destination member. I need to plot a graph by date/time and number of... (5 Replies)
Discussion started by: shog63
5 Replies

6. UNIX for Dummies Questions & Answers

Extract specific lines for graphing

Hello, I have a very large text file with about 2 million lines. Each of the lines starts like.. SNP_12345678 A 1212, 121, 343, ... SNP_12345678 B 4567, 567, 454, ... and so on. I want to extract specific SNPs and plot them by GNUplot or excel. The file is too large to be opened by text... (1 Reply)
Discussion started by: genehunter
1 Replies

7. Solaris

Solaris Performance Monitoring Graphing Tool

Hi All Anyone out there using any graphing tool for Solaris performance data taken either through SAR utility or iosatat, vmstat, nicstat etc. There are a couple on googling like statsview and rrdtool but not sure if anyone is really happy and satisfied with using any of the graphing tool. ... (1 Reply)
Discussion started by: baner_n
1 Replies

8. Shell Programming and Scripting

Using loop reading a file,retrieving data from data base.

Hi All, I am having trouble through, I am reading the input from tab delimited file containing several records, e.g. line1 field1 field2 field3 so on.. line2 field1 field2 field3 so on.. .. .. on the basis of certain fields for each record in input file, I have to retrieve... (1 Reply)
Discussion started by: Sonu4lov
1 Replies

9. Solaris

Solaris open source database and graphing

All, I am looking for some open source database and graphing software to plot some MIB performance data on a Solaris server. Any suggestions would be appreciated. Thanks, Mike (0 Replies)
Discussion started by: bubba112557
0 Replies
Login or Register to Ask a Question