Optimize awk command


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Optimize awk command
# 1  
Old 09-01-2014
Optimize awk command

Code:
WARNING=${1}
CRITICAL=${2}

echo ${OUTPUT} | gawk -F'[=,]' ' {
        V[++n] = $2
        R[n]  = $0
} END {
        for ( i = 1; i <= n; i++) {
                if((V[i] > 0) && (V[i] < V[i+1]))
                        print R[i], ((V[i+1] - V[i]) / V[i]) * 100
                else if ((V[i] > V[i+1]) && (V[i+1] > 0))
                        print R[i], ((V[i+1] - V[i]) / V[i]) * 100
                else if ((V[i] == V[i+1]) && (V[i+1] > 0))
                        print R[i], (V[i+1] - V[i])
                else if ((V[i] == 0) && (V[i+1] > 0))
                        print R[i], (V[i+1] + 1 - V[i] + 1 / 1 * 100)
                else if ((V[i+1] == 0) && (V[i] > 0))
                        print R[i], ((V[i] - V[i+1]) / V[i]) * -100
                else if ((V[i] == 0) && (V[i+1] == 0))
                        print R[i], (V[i] + V[i+1])
                else
                        print R[i]
        }
} ' OFS=, | gawk -F'[=,]' '{if(($4>='"$WARNING"') && ($4<='"$CRITICAL"')) {print $0} }'


so i have the above command which works wonderfully. but i found myself in a situation i didn't expect.

first, i believe the code I bolded above can be incorporated into the preceding awk calculations. any ideas on how to do that?

second, i need to find a way to only show output if and only if the numbers in field 4 ($4), after the calculations are done, is above two interesting numbers.

so the interesting numbers would be something like:

Code:
WARNING=-2
CRITICAL=2

so i want to alert if the numbers is either greater than and/or equal to -2 (negative 2) or greater than or equal to 2 (positive 2).

so in this context, for example, -3 should be considered greater than or equal to -2. and 3 of course should be considered greater than or equal to 2 as well.

any ideas on how to modify the above code to do that, efficiently?

Last edited by SkySmart; 09-01-2014 at 07:03 PM..
# 2  
Old 09-01-2014
Without OUTPUT string it's hard to test but this could replace your two awk scripts:

Code:
WARNING=${1}
CRITICAL=${2}

echo "${OUTPUT}" | gawk -F'[=,]' -vCRIT=${CRITICAL} -vWARN=${WARNING} ' {
        V[++n] = $2
        T[n] = $4
        R[n]  = $0
} END {
        for ( i = 1; i <= n; i++) {
            if(T[i] >= WARN && T[i] <= CRIT) {
                if((V[i] > 0) && (V[i] < V[i+1]))
                        print R[i], ((V[i+1] - V[i]) / V[i]) * 100
                else if ((V[i] > V[i+1]) && (V[i+1] > 0))
                        print R[i], ((V[i+1] - V[i]) / V[i]) * 100
                else if ((V[i] == V[i+1]) && (V[i+1] > 0))
                        print R[i], (V[i+1] - V[i])
                else if ((V[i] == 0) && (V[i+1] > 0))
                        print R[i], (V[i+1] + 1 - V[i] + 1 / 1 * 100)
                else if ((V[i+1] == 0) && (V[i] > 0))
                        print R[i], ((V[i] - V[i+1]) / V[i]) * -100
                else if ((V[i] == 0) && (V[i+1] == 0))
                        print R[i], (V[i] + V[i+1])
                else
                        print R[i]
            }
        }
} ' OFS=,

Note:
  • -3 is not greater than -2
  • It's a good idea to quote the ${OUTPUT} string to stop the shell globbing and param-splitting it.
This User Gave Thanks to Chubler_XL For This Post:
# 3  
Old 09-01-2014
Chubler_XL, thank you so much!

the output looks like this:

Code:
survey=177,value=85.6701,time=[Sep-1-(14:21:02-1409606462-2014)],epoch=1409606462,avg=7.43836,range=120330--120507,-95.4802  survey=177,value=85.6701,time=[Sep-1-(14:21:02-1409606462-2014)],epoch=1409606462,avg=7.43836,range=120330--120507,-95.4802 survey=177,value=85.6701,time=[Sep-1-(14:21:02-1409606462-2014)],epoch=1409606462,avg=7.43836,range=120330--120507,-95.4802

notice the spaces.

and while i'm aware that -3 is not bigger than -2, what i meant to say was, i want to catch values that are less than or equal to -2. in which case, and in the context you are thinking, would mean numbers like -2, -3, -4, -5, -6, etc.
# 4  
Old 09-02-2014
in addition to what Chubler_XL has suggested, you could change the prints in if else conditions as below
Code:
else if ((V[i] == V[i+1]) && (V[i+1] > 0))
        print R[i], (V[i+1] - V[i])
## You could change the print to
print R[i], 0
 
else if ((V[i] == 0) && (V[i+1] > 0))
        print R[i], (V[i+1] + 1 - V[i] + 1 / 1 * 100)
## here, I am not sure what you wanted to achieve, but this can be written
print R[i], (V[i+1] + 101)
 
else if ((V[i+1] == 0) && (V[i] > 0))
        print R[i], ((V[i] - V[i+1]) / V[i]) * -100
## here, again
print R[i], -100
 
else if ((V[i] == 0) && (V[i+1] == 0))
        print R[i], (V[i] + V[i+1])
## here,
print R[i], 0

This User Gave Thanks to SriniShoo For This Post:
# 5  
Old 09-02-2014
Quote:
Originally Posted by SkySmart
Code:
WARNING=${1}
CRITICAL=${2}

echo ${OUTPUT} | gawk -F'[=,]' ' {
        V[++n] = $2
        R[n]  = $0
} END {
        for ( i = 1; i <= n; i++) {
                if((V[i] > 0) && (V[i] < V[i+1]))
                        print R[i], ((V[i+1] - V[i]) / V[i]) * 100
                else if ((V[i] > V[i+1]) && (V[i+1] > 0))
                        print R[i], ((V[i+1] - V[i]) / V[i]) * 100
                else if ((V[i] == V[i+1]) && (V[i+1] > 0))
                        print R[i], (V[i+1] - V[i])
                else if ((V[i] == 0) && (V[i+1] > 0))
                        print R[i], (V[i+1] + 1 - V[i] + 1 / 1 * 100)
                else if ((V[i+1] == 0) && (V[i] > 0))
                        print R[i], ((V[i] - V[i+1]) / V[i]) * -100
                else if ((V[i] == 0) && (V[i+1] == 0))
                        print R[i], (V[i] + V[i+1])
                else
                        print R[i]
        }
} ' OFS=, | gawk -F'[=,]' '{if(($4>='"$WARNING"') && ($4<='"$CRITICAL"')) {print $0} }'


so i have the above command which works wonderfully. but i found myself in a situation i didn't expect.

first, i believe the code I bolded above can be incorporated into the preceding awk calculations. any ideas on how to do that?

second, i need to find a way to only show output if and only if the numbers in field 4 ($4), after the calculations are done, is above two interesting numbers.

so the interesting numbers would be something like:

Code:
WARNING=-2
CRITICAL=2

so i want to alert if the numbers is either greater than and/or equal to -2 (negative 2) or greater than or equal to 2 (positive 2).

so in this context, for example, -3 should be considered greater than or equal to -2. and 3 of course should be considered greater than or equal to 2 as well.

any ideas on how to modify the above code to do that, efficiently?
I'm very confused by this thread. Using the above code with OUTPUT set to:
Code:
survey=177,value=85.6701,time=[Sep-1-(14:21:02-1409606462-2014)],epoch=1409606462,avg=7.43836,range=120330--120507,-95.4802  survey=177,value=85.6701,time=[Sep-1-(14:21:02-1409606462-2014)],epoch=1409606462,avg=7.43836,range=120330--120507,-95.4802 survey=177,value=85.6701,time=[Sep-1-(14:21:02-1409606462-2014)],epoch=1409606462,avg=7.43836,range=120330--120507,-95.4802

(which uses echo to change the two spaces before the second occurrence of survey= into a single space) feeds a single line into gawk. Even if $OUTPUT expanded to contain multiple lines, the echo would convert any adjacent sequence of one or more <space>, <tab>, and <newline> characters into single <space> characters in the data passed in to gawk. Therefore, all of the conditions marked in red above will ALWAYS evaluate to false.

And, you said the above script is working correctly, but you want it to run faster by combining the two invocations of gawk into a single invocation of gawk (or awk), but the description of what you want the code:
Code:
if(($4>='"$WARNING"') && ($4<='"$CRITICAL"')) {print $0}

to do, doesn't even come close to what this code does. If WARNING is set to -2 and CRITICAL is set to 2, this code will print the single input line to your script if and only if $4 in your input (85.6701 in your sample input) is between -2 and 2. Since 85.6701 is not between -2 and 2, your script will produce no output for your sample input. Borrowing from Chubler_XL and taking SriniShoo's evaluation to the next step, your code could be optimized to something like:
Code:
WARNING=${1}
CRITICAL=${2}

echo ${OUTPUT} | gawk -F'[=,]' -v CRIT=${CRITICAL} -v WARN=${WARNING} '
$4 >= WARN && $4 <= CRIT {
        if ($2 > 0)
                print $0, -100
        else if ($2 == 0)
                print $0, 0
        else    print
} ' OFS=,

Or, to match your various English descriptions of how $WARNING and $CRITICAL are to be used, change:
Code:
$4 >= WARN && $4 <= CRIT {

to:
Code:
$4 <= WARN {

or, maybe:
Code:
$4 <= WARN || $4 >= CRIT {

This User Gave Thanks to Don Cragun For This Post:
# 6  
Old 09-02-2014
In addition to the observations made earlier, I think the script could be reduced to something like this, without arrays and enumeration in the END section. With i representing the $2 value V(N) and j representing V(N+1) ..

Code:
WARNING=${1}
CRITICAL=${2}

printf "%s\n" "${OUTPUT}" |
awk -F'[=,]' -v WARN=$WARNING -v CRIT=$CRITICAL '
  NR>1 && ( $4<=WARN || $4>=CRIT ) {
    j=$2
    if      (i>0  && i<=j)  print s, (j?1:-1) * 100 * (j - i) / i
    else if (j>0  && i>j)   print s, 100 * (j - i) / i
    else if (i>=0 && i==j)  print s, 0
    else if (i==0 && j>0)   print s, j - i + 101
    else print s
  }
  {
    i=$2
    s=$0
  }
' OFS=,

# 7  
Old 09-02-2014
I think
Code:
$4*$4 > 4

would satisfy both ($4 < -2 or $4 > 2) conditions...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Optimize multiple awk variable assignment

how can i optimize the following: TOTALRESULT="total1=4 total2=9 total3=89 TMEMORY=1999" TOTAL1=$(echo "${TOTALRESULT}" | egrep "total1=" | awk -F"=" '{print $NF}') TOTAL2=$(echo "${TOTALRESULT}" | egrep "total2=" | awk -F"=" '{print $NF}') TOTAL3=$(echo... (4 Replies)
Discussion started by: SkySmart
4 Replies

2. Shell Programming and Scripting

Help Optimize the Script Further

Hi All, I have written a new script to check for DB space and size of dump log file before it can be imported into a Oracle DB. I'm relatively new to shell scripting. Please help me optimize this script further. (0 Replies)
Discussion started by: narayanv
0 Replies

3. Shell Programming and Scripting

Optimize awk code

sample data.file: 0,mfrh_green_screen,1454687485,383934,/PROD/G/cicsmrch/sys/unikixmain.log,37M,mfrh_green_screen,28961345,0,382962--383934 0,mfrh_green_screen,1454687785,386190,/PROD/G/cicsmrch/sys/unikixmain.log,37M,mfrh_green_screen,29139568,0,383934--386190... (7 Replies)
Discussion started by: SkySmart
7 Replies

4. Shell Programming and Scripting

Looking to optimize code

Hi guys, I feel a bit comfortable now doing bash scripting but I am worried that the way I do it is not optimized and I can do much better as to how I code. e.g. I have a whole line in a file from which I want to extract some values. Right now what I am doing is : STATE=`cat... (5 Replies)
Discussion started by: Junaid Subhani
5 Replies

5. Shell Programming and Scripting

Optimize my mv script

Hello, I'm wondering if there is a quicker way of doing this. Here is my mv script. d=/conversion/program/out cd $d ls $d > /home/tempuser/$$tmp while read line ; do a=`echo $line|cut -c1-5|sed "s/_//g"` b=`echo $line|cut -c16-21` if ;then mkdir... (13 Replies)
Discussion started by: whegra
13 Replies

6. Shell Programming and Scripting

Optimize the nested IF

Hi, I have to assign a value for a varaiable based on a Input. I have written the below code: if then nf=65 elif then nf=46 elif then nf=164 elif then nf=545 elif then nf=56 elif then (3 Replies)
Discussion started by: machomaddy
3 Replies

7. Shell Programming and Scripting

pl help me to Optimize the given code

Pl help to me to write the below code in a simple way ... i suupose to use this code 3 to 4 places in my makefile(gnu) .. **************************************** @for i in $(LIST_A); do \ for j in $(LIST_B); do\ if ;then\ echo "Need to sign"\ echo "List A = $$i , List B =$$j"\ ... (2 Replies)
Discussion started by: pk_arun
2 Replies

8. Shell Programming and Scripting

awk command in script gives error while same awk command at prompt runs fine: Why?

Hello all, Here is what my bash script does: sums number columns, saves the tot in new column, outputs if tot >= threshold val: > cat getnon0file.sh #!/bin/bash this="getnon0file.sh" USAGE=$this" InFile="xyz.38" Min="0.05" # awk '{sum=0; for(n=2; n<=NF; n++){sum+=$n};... (4 Replies)
Discussion started by: catalys
4 Replies

9. Shell Programming and Scripting

can we optimize this command

can we optimize this command ? sed 's#AAAA##g' /study/i.txt | sed '1,2d' | tr -d '\n\' > /study/i1.txt; as here i am using two files ...its overhead..can we optimise to use only 1 file sed 's#AAAA##g' /study/i.txt | sed '1,2d' | tr -d '\n\' > /study/i.txt; keeping them same but it... (9 Replies)
Discussion started by: crackthehit007
9 Replies

10. Shell Programming and Scripting

optimize the script

Hi, I have this following script below. Its searching a log file for 2 string and if found then write the strings to success.txt and If not found write strings to failed.txt . if one found and not other...then write found to success.txt and not found to failed.txt. I want to optimize this... (3 Replies)
Discussion started by: amitrajvarma
3 Replies
Login or Register to Ask a Question