Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Combining awk code into one

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Combining awk code into one
# 1  
Old 09-26-2017
Combining awk code into one
the following code works perfectly for me:
Code:
# AWK 1
gawk -F, '/,'${ThisMonthDOW}' '${ThisMonthMON}' [0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] '${ThisMonthYEA}',/ {
                                                                        if (NF == 10)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10} 
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10} 
                                                                                }
                                                                        else if (NF == 11)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10} 
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10} 
                                                                                }
                                                                }' ${TagFile}.one.${NMON}.mon > total.entries.${NTIME}.mon.dow

## AWK 2
                                                                gawk -F, '/,(Mon|Tue|Wed|Thu|Fri|Sat|Sun) '${ThisMonthMON}' '${ThisMonthDAY}' [0-9][0-9]:[0-9][0-9]:[0-9][0-9] '${ThisMonthYEA}',/ {
                                                                        if (NF == 10)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10} 
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10} 
                                                                                }
                                                                        else if (NF == 11)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10} 
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10} 
                                                                                }
                                                                }' ${TagFile}.one.${NMON}.mon > total.entries.${NTIME}.mon.dom

However, as you can see, it is inefficient as im running awk on a log file 2 different times when I should be able to run it once.

my question is, how can i combine this into one? so that it takes the different log files as arguments. and then searches for the distinct patterns in each of the log files and then outputs the results to the different text files ending in ".dow" and ".dom".
# 2  
Old 09-26-2017
Hi,
Maybe (not tested but permit in gawk) :
Code:
gawk -F, '/,'${ThisMonthDOW}' '${ThisMonthMON}' [0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] '${ThisMonthYEA}',/ {
                                                                        if (NF == 10)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow} 
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                }
                                                                        else if (NF == 11)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                }
                                                                }
/,(Mon|Tue|Wed|Thu|Fri|Sat|Sun) '${ThisMonthMON}' '${ThisMonthDAY}' [0-9][0-9]:[0-9][0-9]:[0-9][0-9] '${ThisMonthYEA}',/ {
                                                                        if (NF == 10)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                }
                                                                        else if (NF == 11)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                }
                                                                }' ${TagFile}.one.${NMON}.mon

Beware: you must empty your files total.entries*

Regards.
This User Gave Thanks to disedorgue For This Post:
SkySmart
# 3  
Old 09-27-2017
this doesnt seem to be working. no files get created.

if it helps, the content of the data file has many lines that look like this:

Code:
0,records_scms1,1506350978,66,DATA.TXT,800pages,records_scms2,6395,0,66--66,3400

# 4  
Old 09-27-2017
Hello SkySmart,

Appreciate that you have shown us what you have tried Smilie. Could you please let us know the sample Input_file and expected output too, so that we could see how we could help here.

Thanks,
R. Singh
# 5  
Old 09-27-2017
my apologies. the code provided my disedorgue works. there was a mistake on my end. sorry for inconvenience.
# 6  
Old 09-27-2017
Rearranging the script a little
Code:
gawk -F, '/,'${ThisMonthDOW}' '${ThisMonthMON}' [0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] '${ThisMonthYEA}',/ {
if (NF == 10)   {split($10, B,"--");
                 if        (B[2] >= B[1])                       {print B[2]-B[1] "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] == 0))        {print "0"       "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] > 0))         {print B[2]      "," $3 "," $10} 
                  else if  (B[2] == B[1])                       {print "0"       "," $3 "," $10} 
                  else if  (B[2] < B[1])                        {print B[1]-B[2] "," $3 "," $10} 
        }
else if (NF == 11) {split($10, B,"--");
                 if        (B[2] >= B[1])                       {print B[2]-B[1] "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] == 0))        {print "0"       "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] > 0))         {print B[2]      "," $3 "," $10} 
                  else if (B[2] == B[1])                        {print "0"       "," $3 "," $10} 
                  else if (B[2] < B[1])                         {print B[1]-B[2] "," $3 "," $10} 
        }
 }' ${TagFile}.one.${NMON}.mon > total.entries.${NTIME}.mon.dow


 gawk -F, '/,(Mon|Tue|Wed|Thu|Fri|Sat|Sun) '${ThisMonthMON}' '${ThisMonthDAY}' [0-9][0-9]:[0-9][0-9]:[0-9][0-9] '${ThisMonthYEA}',/ {
if (NF == 10)   {split($10, B,"--");
                 if        (B[2] >= B[1])                       {print B[2]-B[1] "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] == 0))        {print "0"       "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] > 0))         {print B[2]      "," $3 "," $10} 
                  else if  (B[2] == B[1])                       {print "0"       "," $3 "," $10} 
                  else if  (B[2] < B[1])                        {print B[1]-B[2] "," $3 "," $10} 
        }
else if (NF == 11) {split($10, B,"--");
                 if        (B[2] >= B[1])                       {print B[2]-B[1] "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] == 0))        {print "0"       "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] > 0))         {print B[2]      "," $3 "," $10} 
                  else if  (B[2] == B[1])                       {print "0"       "," $3 "," $10} 
                  else if  (B[2] < B[1])                        {print B[1]-B[2] "," $3 "," $10} 
        }
 }' ${TagFile}.one.${NMON}.mon > total.entries.${NTIME}.mon.dom

shows that there's a decent amount of redundancy. There'd be no difference in the logics had we 10 or 11 fields, nor the output file being ".dow" or ".dom" (which in turn only depends on a small difference of the input line. The split is performed 4 times, 3 times too many.
So, in essence, your pages of script could be reduced to a little bit of logics and a few lines of print statements...

Are you aware that your sample line would not match either regex?
This User Gave Thanks to RudiC For This Post:
SkySmart
# 7  
Old 09-27-2017
Unable to test as neither variable contents nor data samples are available, but could serve as a pointer in a simplification direction:
Code:
awk -F, -vTMW=${ThisMonthDOW} -vTMM=${ThisMonthMON} -vTMD=${ThisMonthDAY} -vTMY=${ThisMonthYEA} -vNTM=${NTIME} '
BEGIN           {FN[1] = "total.entries."NTM".mon.dow"
                 FN[2] = "total.entries."NTM".mon.dom"
                }

                {TYPE = 0
                }
$0 ~ "," TMW " " TMM " [0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] " TMY
                {TYPE = 1
                }
$0 ~ ",(Mon|Tue|Wed|Thu|Fri|Sat|Sun) " TMM " " TMD " [0-9][0-9]:[0-9][0-9]:[0-9][0-9] " TMY
                {TYPE = 2
                }

TYPE && ((NF == 10) || (NF == 11))
                {split($10, B,"--");
                 OUT = abs(B[1] - B[2])
                 if   ((B[1] > B[2]) && (B[2] >= 0))    OUT = B[2]
                 print OUT  "," $3 "," $10 > FN[TYPE]
                } 
 ' ${TagFile}.one.${NMON}.mon

This User Gave Thanks to RudiC For This Post:
SkySmart
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Combining awk Inline and File Code

I've ended up with a small collection of libraries I like to use with awk, but this means I can't do awk -f librarycode.awk '{ program code }' filename because awk will assume that anything after -f is a filename, not code. Is there any way I can do both? (6 Replies)
Discussion started by: Corona688
6 Replies

2. Shell Programming and Scripting

Combining echo and awk

i have a script that has many lines similar to: echo $var | awk -F"--" '{print $2}' as you can see, two commands are being run here. echo and awk. id like to combine this into one awk statement. i tried: awk -F"--" "BEGIN{print $var; print $2}" but i get error messages. (10 Replies)
Discussion started by: SkySmart
10 Replies

3. Shell Programming and Scripting

awk problem - combining awk statements

i have a datafile that has several lines that look like this: 2,dataflow,Sun Mar 17 16:50:01 2013,1363539001,2990,excelsheet,660,mortar,660,4 using the following command: awk -F, '{$3=strftime("%a %b %d %T %Y,%s",$3)}1' OFS=, $DATAFILE | egrep -v "\-OLDISSUES," | ${AWK} "/${MONTH} ${DAY}... (7 Replies)
Discussion started by: SkySmart
7 Replies

4. Shell Programming and Scripting

How to combining awk commands?

I can achieve two tasks with 2 different awk commands: 1) awk -F";;WORD" '{print $2}' file | sed '/^$/d' #to find surface_word 2) awk -F"bw:|gloss:" '// {print $2}' file | sed '/\//!d; s:/*+*: + :g; s:^+::; s: *+ *$::;' #to find segmentation of surface_word Number 1) finds surface_word... (7 Replies)
Discussion started by: Viernes
7 Replies

5. Shell Programming and Scripting

combining awk and sed

Hi experts, I have a requirement, In which I need to display the first and last line of a zip file where the line starts with "L". I've writen the code like below using sed and awk. gunzip -c 20110203.1104.gz | awk '$1 ~ "^L" {print substr($0,178,15)}' | sed -n '1p;$p' Is it possible to do it... (8 Replies)
Discussion started by: senthil.ak
8 Replies

6. Shell Programming and Scripting

Combining awk statements

I have a pretty simple script below: #!/bin/sh for i in *.cfg do temp=`awk '/^InputDirectory=/' ${i}` input_dir=`echo ${temp} | awk '{ print substr( $0, 16) }'` echo ${input_dir} done As you can see its opening each cfg file and searching for the line that has "InputDirectory="... (3 Replies)
Discussion started by: ssbsts
3 Replies

7. Shell Programming and Scripting

Combining two awk scripts

I have a file like this consisting of blocks separated by > of two number X and T > 10 0 13 5.92346 16 10.3106 19 13.9672 22 16.9838 25 19.4407 28 21.4705 31 23.1547 34 24.6813 37 26.0695 40 27.3611 43 28.631 46 29.8366 49 30.9858 52 32.0934 55 33.1458 (6 Replies)
Discussion started by: kristinu
6 Replies

8. UNIX for Dummies Questions & Answers

Combining awk tests

It would be convenient to be able to combine awk tests. For example, suppose that I do this query: awk '$1 != "Bob" || $1 != "Linda" {print $2}' datafileIs there a reasonable way to combine the conditions into a single statement? For example, in egrep, I can do: egrep -v "Bob|Linda"... (4 Replies)
Discussion started by: treesloth
4 Replies

9. Shell Programming and Scripting

combining fields in awk

I am using: ps -A -o command,%cpu to get process and cpu usage figures. I want to use awk to split up the columns it returns. If I use: awk '{print "Process: "$1"\nCPU Usage: "$NF"\n"}' the $NF will get me the value in the last column, but if there is more than one word in the... (2 Replies)
Discussion started by: json4639
2 Replies

10. Shell Programming and Scripting

help combining lines in awk

I seem to have gotten myself in over my head on this one. I need help combining lines together. I have a text file containing 24,000 lines (exactly why I need awk) due to bad formatting it has separated the lines (ideally it should be 12,000 lines total). Example of file: ... (2 Replies)
Discussion started by: blueheed
2 Replies
Login or Register to Ask a Question