Awking custom output


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Awking custom output
# 1  
Old 01-08-2017
Awking custom output

i have data that can look like this:

Code:
echo "Master_Item_Service_is_down=0_njava_lang_NoClassDefFoundError=0_njava_lang_OutOfMemoryError=1_nemxCommonAppInitialization__Error_while_initializing=0_nINFO__Stopping_Coyote_HTTP_1_1_on_http_8080=7_nThe_file_or_directory_is_corrupted_and_unreadable=0_n"

or

Code:
echo "Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=1 emxCommonAppInitialization__Error_while_initializing=0 INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=7 The_file_or_directory_is_corrupted_and_unreadable=0"

or

Code:
_error_=0-- _fatal_=0-- _panic_=0-- _fault_=0

I need to grab the number that comes right after the equal sign "=" for each of the patterns and after getting all the numbers, I want add them up to get the total.

so in the above scenario, there is a total of 8 errors

im looking for a solution that will take into account scenarios of both output. im hoping awk can be used for this.

Last edited by SkySmart; 01-08-2017 at 02:44 AM..
# 2  
Old 01-08-2017
Try:
Code:
awk -F= '{t=0; for(i=2; i<=NF; i++) t+=$i; print t}'

This User Gave Thanks to Scrutinizer For This Post:
# 3  
Old 01-08-2017
thank you for your code Scrutinizer. it works perfectly!

one more request.

the actual datafile looks like this:

Code:
0,tomcat_logcheck,1483897732,240,/opt/apps/plmviewcafe/logs/catalina.out,16K,tomcat_logcheck,14560,Master_Item_Service_is_down=0_njava_lang_NoClassDefFoundError=0_njava_lang_OutOfMemoryError=0_nemxCommonAppInitialization__Error_while_initializing=0_nINFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0_nThe_file_or_directory_is_corrupted_and_unreadable=0_n,174--240,181
0,tomcat_logcheck,1483898023,309,/opt/apps/plmviewcafe/logs/catalina.out,20K,tomcat_logcheck,19277,Master_Item_Service_is_down=0_njava_lang_NoClassDefFoundError=0_njava_lang_OutOfMemoryError=0_nemxCommonAppInitialization__Error_while_initializing=0_nINFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0_nThe_file_or_directory_is_corrupted_and_unreadable=0_n,240--309,25

and i run the following awk command on the data:

Code:
gawk -v SEARCHPATT="${SEARCHPATT}" -v ADDISTR="${INCEXCSTR}" -F, '/,'"${VALFOUND}"',/,0 {A=strftime("%a %b %d %T %Y,%s",$3);if((NF == 13) && (A ~ ADDISTR) && (A ~ SEARCHPATT)) {print $12"-"$3"_0""-" $13"----"A} else if ((NF == 14) && (A ~ ADDISTR) && (A ~ SEARCHPATT)) {print $12"-"$3"_0""-" $13"----"A} else if ((NF == 10) && (A ~ ADDISTR) && (A ~ SEARCHPATT)) {print $9"-"$3"_"$10"----"A} else if ((NF == 11) && (A ~ SEARCHPATT)) {print $9"-"$3"_"$10"----"A} }' datafile.txt | awk -F"----" '{print $1}'

which normally produces an expected output similar to this:

Code:
0-1424534260_8--8
0-1424534560_8--8
0-1424534860_8--8
0-1424535160_8--8
0-1424535460_8--8
0-1424535760_8--8
0-1424536060_8--8
0-1424536360_8--8

i get the above output only if the 9th field contains a value. but if the 9th field contains the original values I posted in this thread, "Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=1 emxCommonAppInitialization__Error_while_initializing=0 ", i get output similar to the following, which is not what i want.:

Code:
emxCommonAppInitialization__Error_while_initializing=0-- INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0-- java_lang_NoClassDefFoundError=0-- java_lang_OutOfMemoryError=0-- Master_Item_Service_is_down=0-- The_file_or_directory_is_corrupted_and_unreadable=0-1413868231_12043--12043


so i would like to incorporate your command into my original command that i pasted in this post, so that it adds up all the values in the 9th frield and then shows the expected output:

Code:
0-1413868231_12043--12043
.....


with the bolded being the total of all the values in that 9th column. sorry if i just made this too complicated.
# 4  
Old 01-09-2017
Your expected output doesn't really fit what your incredibly overcomplex script will produce when applied to your input sample. Which doesn't necessarily help us help you. Nor does the missing context like the undefined shell variables.

You seem to want to print $(NF-D)"-"$3"_"$(NF-E)"----"A with D equals 1 or 2 and E equals 0 or 1 depending on the field count of the line. How about considering boiling down above to something like
Code:
$0 ~ "," VF "," {L = 1
                }

!L              {next
                }

A ~ SEARCHPATT  {D == 0
                 if (NF == 10 || NF == 13)      {D = 1 
                                                 E = 0
                                                }
                 if (NF == 11 || NF == 14)      {D = 2 
                                                 E = 1
                                                }
                 F = (NF == 11 || A ~ ADDISTR)

                 if (D && F)                    print $(NF-D)"-"$3"_"$(NF-E)"----"A
                }

Why do you print the A variable if immediately afterwards remove it again?


For your $9 problem, you might want to try

Code:
{TOT = 0; for (n = split ($(NF-D), T, "="); n>1; n--) {sub (/_.*/, _, T[n]); TOT += T[n]};

and print TOT in lieu of the resp. field.

Last edited by RudiC; 01-09-2017 at 07:35 AM..
This User Gave Thanks to RudiC For This Post:
# 5  
Old 01-11-2017
Quote:
Originally Posted by RudiC
Your expected output doesn't really fit what your incredibly overcomplex script will produce when applied to your input sample. Which doesn't necessarily help us help you. Nor does the missing context like the undefined shell variables.

You seem to want to print $(NF-D)"-"$3"_"$(NF-E)"----"A with D equals 1 or 2 and E equals 0 or 1 depending on the field count of the line. How about considering boiling down above to something like
Code:
$0 ~ "," VF "," {L = 1
                }

!L              {next
                }

A ~ SEARCHPATT  {D == 0
                 if (NF == 10 || NF == 13)      {D = 1 
                                                 E = 0
                                                }
                 if (NF == 11 || NF == 14)      {D = 2 
                                                 E = 1
                                                }
                 F = (NF == 11 || A ~ ADDISTR)

                 if (D && F)                    print $(NF-D)"-"$3"_"$(NF-E)"----"A
                }

Why do you print the A variable if immediately afterwards remove it again?


For your $9 problem, you might want to try

Code:
{TOT = 0; for (n = split ($(NF-D), T, "="); n>1; n--) {sub (/_.*/, _, T[n]); TOT += T[n]};

and print TOT in lieu of the resp. field.
im trying to boil down the command as you suggested, but im running into issues:

Code:
gawk -v SEARCHPATT="(Wed|Tue)" -v ADDISTR="Mon|Tue|Wed|Thu|Fri|Sat|Sun" -F, '/,1484023642,/,0 {A=strftime("%a %b %d %T %Y,%s",$3); {$0 ~ "," VF "," {L = 1
                }

!L              {next
                }

A ~ SEARCHPATT  {D == 0
                 if (NF == 10 || NF == 13)      {D = 1 
                                                 E = 0
                                                }
                 if (NF == 11 || NF == 14)      {D = 2 
                                                 E = 1
                                                }
                 F = (NF == 11 || A ~ ADDISTR)
                 if (D && F)                    print $(NF-D)"-"$3"_"$(NF-E)"----"A
}
}' datafile.txt

errors I get:

Code:
gawk: /,1484023642,/,0 {A=strftime("%a %b %d %T %Y,%s",$3); {$0 ~ "," VF "," {L = 1
gawk:                                                                        ^ syntax error
gawk: cmd. line:3: !L              {next
gawk: cmd. line:3:                 ^ syntax error
gawk: cmd. line:6: A ~ SEARCHPATT  {D == 0
gawk: cmd. line:6:                 ^ syntax error
gawk: cmd. line:17: }
gawk: cmd. line:17:  ^ unexpected newline or end of string
[mojomo@pgphxplmap004 ~]$

# 6  
Old 01-11-2017
Of course. You messed up the pattern {action} pair structure of awk. Try (untested)

Code:
awk -F, -v SEARCHPATT="(Wed|Tue)" -v ADDISTR="Mon|Tue|Wed|Thu|Fri|Sat|Sun" -vVF="$VALFOUND"
BEGIN           {D[10] = D[13] = 1
                 D[11] = D[14] = 2
                }

$0 ~ "," VF "," {L = 1                                  # start output only if VALFOUND is matched
                }

!L              {next                                   # skip line if NOT VALFOUND
                }

                {A = strftime("%a %b %d %T %Y,%s",$3)
                }

A ~ SEARCHPATT &&
NF in D         {TOT = 0
                 for (n = split ($(NF-D[NF]), T, "="); n>1; n--)    {sub (/_.*/, _, T[n]); TOT += T[n]}

                 if (NF == 11 || A ~ ADDISTR)   print TOT "-" $3 "_" $(NF-D[NF]+1) "----" A
                }
}' datafile.txt

EDIT: Actually, looking at it again, there's another logic flaw: if A ~ "(Wed|Tue)" it will ALWAYS match Mon|Tue|Wed|Thu|Fri|Sat|Sun, so the if in front of the print is pointless.

Last edited by RudiC; 01-11-2017 at 03:56 PM..
This User Gave Thanks to RudiC For This Post:
# 7  
Old 01-11-2017
Quote:
Originally Posted by RudiC
Of course. You messed up the pattern {action} pair structure of awk. Try (untested)

Code:
awk -F, -v SEARCHPATT="(Wed|Tue)" -v ADDISTR="Mon|Tue|Wed|Thu|Fri|Sat|Sun" -vVF="$VALFOUND"
BEGIN     '{      {D[10] = D[13] = 1
                 D[11] = D[14] = 2
                }

$0 ~ "," VF "," {L = 1                                  # start output only if VALFOUND is matched
                }

!L              {next                                   # skip line if NOT VALFOUND
                }

                {A = strftime("%a %b %d %T %Y,%s",$3)
                }

A ~ SEARCHPATT &&
NF in D         {TOT = 0
                 for (n = split ($(NF-D[NF]), T, "="); n>1; n--)    {sub (/_.*/, _, T[n]); TOT += T[n]}

                 if (NF == 11 || A ~ ADDISTR)   print TOT "-" $3 "_" $(NF-D[NF]+1) "----" A
                }
}' datafile.txt

EDIT: Actually, looking at it again, there's another logic flaw: if A ~ "(Wed|Tue)" it will ALWAYS match Mon|Tue|Wed|Thu|Fri|Sat|Sun, so the if in front of the print is pointless.
im probably missing something here because when i run this script, it still bails out:
Code:
# cat ./hen:   
#!/bin/sh

awk -F, -v SEARCHPATT="(Wed|Tue)" -v ADDISTR="Mon|Tue|Wed|Thu|Fri|Sat|Sun" -vVF="$VALFOUND"
BEGIN   '{           {D[10] = D[13] = 1
                 D[11] = D[14] = 2
                }

$0 ~ "," VF "," {L = 1                                  # start output only if VALFOUND is matched
                }

!L              {next                                   # skip line if NOT VALFOUND
                }

                {A = strftime("%a %b %d %T %Y,%s",$3)
                }

A ~ SEARCHPATT &&
NF in D         {TOT = 0
                 for (n = split ($(NF-D[NF]), T, "="); n>1; n--)    {sub (/_.*/, _, T[n]); TOT += T[n]}

                 if (NF == 11 || A ~ ADDISTR)   print TOT "-" $3 "_" $(NF-D[NF]+1) "----" A
                }
}' tomcatlogcheck

# ./hen 
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
POSIX options:          GNU long options:
        -f progfile             --file=progfile
        -F fs                   --field-separator=fs
        -v var=val              --assign=var=val
        -m[fr] val
        -O                      --optimize
        -W compat               --compat
        -W copyleft             --copyleft
        -W copyright            --copyright
        -W dump-variables[=file]        --dump-variables[=file]
        -W exec=file            --exec=file
        -W gen-po               --gen-po
        -W help                 --help
        -W lint[=fatal]         --lint[=fatal]
        -W lint-old             --lint-old
        -W non-decimal-data     --non-decimal-data
        -W profile[=file]       --profile[=file]
        -W posix                --posix
        -W re-interval          --re-interval
        -W source=program-text  --source=program-text
        -W traditional          --traditional
        -W usage                --usage
        -W use-lc-numeric       --use-lc-numeric
        -W version              --version

To report bugs, see node `Bugs' in `gawk.info', which is
section `Reporting Problems and Bugs' in the printed version.

gawk is a pattern scanning and processing language.
By default it reads standard input and writes standard output.

Examples:
        gawk '{ sum += $1 }; END { print sum }' file
        gawk -F: '{ print $1 }' /etc/passwd
./hen: line 4: BEGIN: command not found
./hen: line 5: D[11]: command not found
./hen: line 6: syntax error near unexpected token `}'
./hen: line 6: `                }'


Last edited by SkySmart; 01-11-2017 at 06:56 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Awking binary data

i have a binary data that has some text in it. what i want to do is, i want to grab just piece of information from the binary. but when i run my awk on it, it returns nothing. awk -F"MYINFO=" '{print $2}' mybinary however, when i install gawk, then try again, it works. i would prefer... (1 Reply)
Discussion started by: SkySmart
1 Replies

2. Shell Programming and Scripting

Custom wget output

The below hides the messy commands of wget #!/bin/bash cd 'C:\Users\cmccabe\Desktop\wget' wget -O getCSV.txt http://172.24.188.113/data/getCSV.csv progressfilt () { local flag=false c count cr=$'\r' nl=$'\n' while IFS='' read -d '' -rn 1 c do if $flag ... (5 Replies)
Discussion started by: cmccabe
5 Replies

3. Shell Programming and Scripting

Script to output custom characters in different order

Need a script to print a set of characters in different combination. What's the key to accomplish this. e.g charset: "Abcdefghij1" without quotes. block 1: "Abcd" block 2: "efg" block 3: "hij1" I need this script to change the order of the characters and print it to stdout e.g print out... (3 Replies)
Discussion started by: zaonline
3 Replies

4. Red Hat

Custom output on FIND

I have a file, ENV.doc somewhere in my home directory. I want to know where the file is located in my sub directories using FIND. But, I want to display only the relative path along with the file name. Thanks, (6 Replies)
Discussion started by: ashok.g
6 Replies

5. HP-UX

Output of Custom package scripts to terminal

Hi, I am doing some testing with creation of depots on HP-UX systems (11.11). Want to display some echo statements based on the processing during checkinstall, pre & postinstall scripts on the terminal. The echo statements are getting directed to /var/adm/sw/swagent.log I want to display... (7 Replies)
Discussion started by: vibhor_agarwali
7 Replies

6. Shell Programming and Scripting

awking two columns based on symbols

The input file has 3 columns. the first column with low values second with bigger.If the symbol is - in third column the numbers have to change the least in column1 and highest in col2. Input col1 col2 col3 1 2 + 2 3 - 3 4 + 5 6 - Output col1 col2 col3 1 2 + 3 2 - 3 4 + 6 5 - The... (2 Replies)
Discussion started by: stateperl
2 Replies

7. Shell Programming and Scripting

awking two columns

Hey ppl I have two columns with random values. i need to insert the 1st row of the first column with the highest number of the two rows in the first column and vice versa. some thing like this. I'm sorry If my question is unclear...:rolleyes: input col1 col2 12...11 11...14 34...45... (11 Replies)
Discussion started by: sophiesophie
11 Replies

8. Shell Programming and Scripting

Help with awking

Hi... How does awk or sed or even grep extract the following string out of my text file?. '"${ETL_VW_SCHEMA}"'.IATA_STN_TZ_UTC_LCL_CONV_CV '"${ETL_VW_SCHEMA}"'.CO '"${ETL_VW_SCHEMA}"'.TEST_CO I just need to extract the second entry which means I need to let awk know I am just searching for... (5 Replies)
Discussion started by: anduzzi
5 Replies

9. Shell Programming and Scripting

Awking

Could someone find out wht exactly is goin wrong in the following awk: awk '/${EDW_DB_SCHEMA}.WRKR/ || !/otable/&&/${EDW_DB_SCHEMA}.WRKR/ || !/db-ter-load-data/&&/${EDW_DB_SCHEMA}.WRKR/' <my_graph>.ksh Basically, I am trying to achieve: Find out the occurence of WRKR table in <my_graph>.ksh... (3 Replies)
Discussion started by: anduzzi
3 Replies

10. Shell Programming and Scripting

Awking!! Printing decimal output is struck

Hi friends, I have a small problem with AWK. I am not able to print decimal values! :confused: below is my code: #! /bin/awk -f awk BEGIN{printf("%d",123)}; -> This prints the integer properly. x=111 awk BEGIN{printf("%d",x)}; -> This doesnt print! :( Please help me solve this. It... (4 Replies)
Discussion started by: divzz
4 Replies
Login or Register to Ask a Question