Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Awking custom output

Shell Programming and Scripting


Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 01-08-2017
SkySmart SkySmart is offline
Registered User
 
Join Date: Dec 2006
Last Activity: 23 February 2017, 10:13 PM EST
Posts: 845
Thanks: 670
Thanked 3 Times in 3 Posts
Awking custom output

i have data that can look like this:


Code:
echo "Master_Item_Service_is_down=0_njava_lang_NoClassDefFoundError=0_njava_lang_OutOfMemoryError=1_nemxCommonAppInitialization__Error_while_initializing=0_nINFO__Stopping_Coyote_HTTP_1_1_on_http_8080=7_nThe_file_or_directory_is_corrupted_and_unreadable=0_n"

or


Code:
echo "Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=1 emxCommonAppInitialization__Error_while_initializing=0 INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=7 The_file_or_directory_is_corrupted_and_unreadable=0"

or


Code:
_error_=0-- _fatal_=0-- _panic_=0-- _fault_=0

I need to grab the number that comes right after the equal sign "=" for each of the patterns and after getting all the numbers, I want add them up to get the total.

so in the above scenario, there is a total of 8 errors

im looking for a solution that will take into account scenarios of both output. im hoping awk can be used for this.

Last edited by SkySmart; 01-08-2017 at 02:44 AM..
Sponsored Links
    #2  
Old Unix and Linux 01-08-2017
Scrutinizer's Unix or Linux Image
Scrutinizer Scrutinizer is online now Forum Staff  
Moderator
 
Join Date: Nov 2008
Last Activity: 24 February 2017, 2:53 AM EST
Location: Amsterdam
Posts: 11,248
Thanks: 467
Thanked 3,200 Times in 2,827 Posts
Try:

Code:
awk -F= '{t=0; for(i=2; i<=NF; i++) t+=$i; print t}'

The Following User Says Thank You to Scrutinizer For This Useful Post:
SkySmart (01-08-2017)
Sponsored Links
    #3  
Old Unix and Linux 01-08-2017
SkySmart SkySmart is offline
Registered User
 
Join Date: Dec 2006
Last Activity: 23 February 2017, 10:13 PM EST
Posts: 845
Thanks: 670
Thanked 3 Times in 3 Posts
thank you for your code Scrutinizer. it works perfectly!

one more request.

the actual datafile looks like this:


Code:
0,tomcat_logcheck,1483897732,240,/opt/apps/plmviewcafe/logs/catalina.out,16K,tomcat_logcheck,14560,Master_Item_Service_is_down=0_njava_lang_NoClassDefFoundError=0_njava_lang_OutOfMemoryError=0_nemxCommonAppInitialization__Error_while_initializing=0_nINFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0_nThe_file_or_directory_is_corrupted_and_unreadable=0_n,174--240,181
0,tomcat_logcheck,1483898023,309,/opt/apps/plmviewcafe/logs/catalina.out,20K,tomcat_logcheck,19277,Master_Item_Service_is_down=0_njava_lang_NoClassDefFoundError=0_njava_lang_OutOfMemoryError=0_nemxCommonAppInitialization__Error_while_initializing=0_nINFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0_nThe_file_or_directory_is_corrupted_and_unreadable=0_n,240--309,25

and i run the following awk command on the data:


Code:
gawk -v SEARCHPATT="${SEARCHPATT}" -v ADDISTR="${INCEXCSTR}" -F, '/,'"${VALFOUND}"',/,0 {A=strftime("%a %b %d %T %Y,%s",$3);if((NF == 13) && (A ~ ADDISTR) && (A ~ SEARCHPATT)) {print $12"-"$3"_0""-" $13"----"A} else if ((NF == 14) && (A ~ ADDISTR) && (A ~ SEARCHPATT)) {print $12"-"$3"_0""-" $13"----"A} else if ((NF == 10) && (A ~ ADDISTR) && (A ~ SEARCHPATT)) {print $9"-"$3"_"$10"----"A} else if ((NF == 11) && (A ~ SEARCHPATT)) {print $9"-"$3"_"$10"----"A} }' datafile.txt | awk -F"----" '{print $1}'

which normally produces an expected output similar to this:


Code:
0-1424534260_8--8
0-1424534560_8--8
0-1424534860_8--8
0-1424535160_8--8
0-1424535460_8--8
0-1424535760_8--8
0-1424536060_8--8
0-1424536360_8--8

i get the above output only if the 9th field contains a value. but if the 9th field contains the original values I posted in this thread, "Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=1 emxCommonAppInitialization__Error_while_initializing=0 ", i get output similar to the following, which is not what i want.:


Code:
emxCommonAppInitialization__Error_while_initializing=0-- INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0-- java_lang_NoClassDefFoundError=0-- java_lang_OutOfMemoryError=0-- Master_Item_Service_is_down=0-- The_file_or_directory_is_corrupted_and_unreadable=0-1413868231_12043--12043


so i would like to incorporate your command into my original command that i pasted in this post, so that it adds up all the values in the 9th frield and then shows the expected output:


Code:
0-1413868231_12043--12043
.....


with the bolded being the total of all the values in that 9th column. sorry if i just made this too complicated.
    #4  
Old Unix and Linux 01-09-2017
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 23 February 2017, 4:37 PM EST
Location: Aachen, Germany
Posts: 10,329
Thanks: 244
Thanked 3,153 Times in 2,918 Posts
Your expected output doesn't really fit what your incredibly overcomplex script will produce when applied to your input sample. Which doesn't necessarily help us help you. Nor does the missing context like the undefined shell variables.

You seem to want to print $(NF-D)"-"$3"_"$(NF-E)"----"A with D equals 1 or 2 and E equals 0 or 1 depending on the field count of the line. How about considering boiling down above to something like

Code:
$0 ~ "," VF "," {L = 1
                }

!L              {next
                }

A ~ SEARCHPATT  {D == 0
                 if (NF == 10 || NF == 13)      {D = 1 
                                                 E = 0
                                                }
                 if (NF == 11 || NF == 14)      {D = 2 
                                                 E = 1
                                                }
                 F = (NF == 11 || A ~ ADDISTR)

                 if (D && F)                    print $(NF-D)"-"$3"_"$(NF-E)"----"A
                }

Why do you print the A variable if immediately afterwards remove it again?


For your $9 problem, you might want to try


Code:
{TOT = 0; for (n = split ($(NF-D), T, "="); n>1; n--) {sub (/_.*/, _, T[n]); TOT += T[n]};

and print TOT in lieu of the resp. field.

Last edited by RudiC; 01-09-2017 at 07:35 AM..
The Following User Says Thank You to RudiC For This Useful Post:
SkySmart (01-09-2017)
Sponsored Links
    #5  
Old Unix and Linux 01-11-2017
SkySmart SkySmart is offline
Registered User
 
Join Date: Dec 2006
Last Activity: 23 February 2017, 10:13 PM EST
Posts: 845
Thanks: 670
Thanked 3 Times in 3 Posts
Quote:
Originally Posted by RudiC View Post
Your expected output doesn't really fit what your incredibly overcomplex script will produce when applied to your input sample. Which doesn't necessarily help us help you. Nor does the missing context like the undefined shell variables.

You seem to want to print $(NF-D)"-"$3"_"$(NF-E)"----"A with D equals 1 or 2 and E equals 0 or 1 depending on the field count of the line. How about considering boiling down above to something like

Code:
$0 ~ "," VF "," {L = 1
                }

!L              {next
                }

A ~ SEARCHPATT  {D == 0
                 if (NF == 10 || NF == 13)      {D = 1 
                                                 E = 0
                                                }
                 if (NF == 11 || NF == 14)      {D = 2 
                                                 E = 1
                                                }
                 F = (NF == 11 || A ~ ADDISTR)

                 if (D && F)                    print $(NF-D)"-"$3"_"$(NF-E)"----"A
                }

Why do you print the A variable if immediately afterwards remove it again?


For your $9 problem, you might want to try


Code:
{TOT = 0; for (n = split ($(NF-D), T, "="); n>1; n--) {sub (/_.*/, _, T[n]); TOT += T[n]};

and print TOT in lieu of the resp. field.
im trying to boil down the command as you suggested, but im running into issues:


Code:
gawk -v SEARCHPATT="(Wed|Tue)" -v ADDISTR="Mon|Tue|Wed|Thu|Fri|Sat|Sun" -F, '/,1484023642,/,0 {A=strftime("%a %b %d %T %Y,%s",$3); {$0 ~ "," VF "," {L = 1
                }

!L              {next
                }

A ~ SEARCHPATT  {D == 0
                 if (NF == 10 || NF == 13)      {D = 1 
                                                 E = 0
                                                }
                 if (NF == 11 || NF == 14)      {D = 2 
                                                 E = 1
                                                }
                 F = (NF == 11 || A ~ ADDISTR)
                 if (D && F)                    print $(NF-D)"-"$3"_"$(NF-E)"----"A
}
}' datafile.txt

errors I get:


Code:
gawk: /,1484023642,/,0 {A=strftime("%a %b %d %T %Y,%s",$3); {$0 ~ "," VF "," {L = 1
gawk:                                                                        ^ syntax error
gawk: cmd. line:3: !L              {next
gawk: cmd. line:3:                 ^ syntax error
gawk: cmd. line:6: A ~ SEARCHPATT  {D == 0
gawk: cmd. line:6:                 ^ syntax error
gawk: cmd. line:17: }
gawk: cmd. line:17:  ^ unexpected newline or end of string
[mojomo@pgphxplmap004 ~]$

Sponsored Links
    #6  
Old Unix and Linux 01-11-2017
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 23 February 2017, 4:37 PM EST
Location: Aachen, Germany
Posts: 10,329
Thanks: 244
Thanked 3,153 Times in 2,918 Posts
Of course. You messed up the pattern {action} pair structure of awk. Try (untested)


Code:
awk -F, -v SEARCHPATT="(Wed|Tue)" -v ADDISTR="Mon|Tue|Wed|Thu|Fri|Sat|Sun" -vVF="$VALFOUND"
BEGIN           {D[10] = D[13] = 1
                 D[11] = D[14] = 2
                }

$0 ~ "," VF "," {L = 1                                  # start output only if VALFOUND is matched
                }

!L              {next                                   # skip line if NOT VALFOUND
                }

                {A = strftime("%a %b %d %T %Y,%s",$3)
                }

A ~ SEARCHPATT &&
NF in D         {TOT = 0
                 for (n = split ($(NF-D[NF]), T, "="); n>1; n--)    {sub (/_.*/, _, T[n]); TOT += T[n]}

                 if (NF == 11 || A ~ ADDISTR)   print TOT "-" $3 "_" $(NF-D[NF]+1) "----" A
                }
}' datafile.txt

EDIT: Actually, looking at it again, there's another logic flaw: if A ~ "(Wed|Tue)" it will ALWAYS match Mon|Tue|Wed|Thu|Fri|Sat|Sun, so the if in front of the print is pointless.

Last edited by RudiC; 01-11-2017 at 03:56 PM..
The Following User Says Thank You to RudiC For This Useful Post:
SkySmart (01-11-2017)
Sponsored Links
    #7  
Old Unix and Linux 01-11-2017
SkySmart SkySmart is offline
Registered User
 
Join Date: Dec 2006
Last Activity: 23 February 2017, 10:13 PM EST
Posts: 845
Thanks: 670
Thanked 3 Times in 3 Posts
Quote:
Originally Posted by RudiC View Post
Of course. You messed up the pattern {action} pair structure of awk. Try (untested)


Code:
awk -F, -v SEARCHPATT="(Wed|Tue)" -v ADDISTR="Mon|Tue|Wed|Thu|Fri|Sat|Sun" -vVF="$VALFOUND"
BEGIN     '{      {D[10] = D[13] = 1
                 D[11] = D[14] = 2
                }

$0 ~ "," VF "," {L = 1                                  # start output only if VALFOUND is matched
                }

!L              {next                                   # skip line if NOT VALFOUND
                }

                {A = strftime("%a %b %d %T %Y,%s",$3)
                }

A ~ SEARCHPATT &&
NF in D         {TOT = 0
                 for (n = split ($(NF-D[NF]), T, "="); n>1; n--)    {sub (/_.*/, _, T[n]); TOT += T[n]}

                 if (NF == 11 || A ~ ADDISTR)   print TOT "-" $3 "_" $(NF-D[NF]+1) "----" A
                }
}' datafile.txt

EDIT: Actually, looking at it again, there's another logic flaw: if A ~ "(Wed|Tue)" it will ALWAYS match Mon|Tue|Wed|Thu|Fri|Sat|Sun, so the if in front of the print is pointless.
im probably missing something here because when i run this script, it still bails out:

Code:
# cat ./hen:   
#!/bin/sh

awk -F, -v SEARCHPATT="(Wed|Tue)" -v ADDISTR="Mon|Tue|Wed|Thu|Fri|Sat|Sun" -vVF="$VALFOUND"
BEGIN   '{           {D[10] = D[13] = 1
                 D[11] = D[14] = 2
                }

$0 ~ "," VF "," {L = 1                                  # start output only if VALFOUND is matched
                }

!L              {next                                   # skip line if NOT VALFOUND
                }

                {A = strftime("%a %b %d %T %Y,%s",$3)
                }

A ~ SEARCHPATT &&
NF in D         {TOT = 0
                 for (n = split ($(NF-D[NF]), T, "="); n>1; n--)    {sub (/_.*/, _, T[n]); TOT += T[n]}

                 if (NF == 11 || A ~ ADDISTR)   print TOT "-" $3 "_" $(NF-D[NF]+1) "----" A
                }
}' tomcatlogcheck

# ./hen 
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
POSIX options:          GNU long options:
        -f progfile             --file=progfile
        -F fs                   --field-separator=fs
        -v var=val              --assign=var=val
        -m[fr] val
        -O                      --optimize
        -W compat               --compat
        -W copyleft             --copyleft
        -W copyright            --copyright
        -W dump-variables[=file]        --dump-variables[=file]
        -W exec=file            --exec=file
        -W gen-po               --gen-po
        -W help                 --help
        -W lint[=fatal]         --lint[=fatal]
        -W lint-old             --lint-old
        -W non-decimal-data     --non-decimal-data
        -W profile[=file]       --profile[=file]
        -W posix                --posix
        -W re-interval          --re-interval
        -W source=program-text  --source=program-text
        -W traditional          --traditional
        -W usage                --usage
        -W use-lc-numeric       --use-lc-numeric
        -W version              --version

To report bugs, see node `Bugs' in `gawk.info', which is
section `Reporting Problems and Bugs' in the printed version.

gawk is a pattern scanning and processing language.
By default it reads standard input and writes standard output.

Examples:
        gawk '{ sum += $1 }; END { print sum }' file
        gawk -F: '{ print $1 }' /etc/passwd
./hen: line 4: BEGIN: command not found
./hen: line 5: D[11]: command not found
./hen: line 6: syntax error near unexpected token `}'
./hen: line 6: `                }'


Last edited by SkySmart; 01-11-2017 at 06:56 PM..
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Custom wget output cmccabe Shell Programming and Scripting 5 04-30-2015 06:00 PM
Script to output custom characters in different order zaonline Shell Programming and Scripting 3 01-30-2012 07:19 PM
Custom output on FIND ashok.g Red Hat 6 12-02-2009 05:06 AM
Output of Custom package scripts to terminal vibhor_agarwali HP-UX 7 03-30-2009 03:31 AM
Awking!! Printing decimal output is struck divzz Shell Programming and Scripting 4 05-28-2008 09:02 AM



All times are GMT -4. The time now is 04:02 AM.