Read in Multiple log files and output selected variables and values to cvs file

06-02-2016

Registered User

51, 0

Join Date: Jul 2015

Last Activity: 8 December 2019, 5:10 PM EST

Posts: 51

Thanks Given: 21

Thanked 0 Times in 0 Posts

Read in Multiple log files and output selected variables and values to cvs file

Quote:

1. The OS is AIX
2. I am using bash shell
3. The format of text files is based on generated sas programs that produce

sas log files. The sas programmers sometimes add the following
parameters to their code Memsize, and a directory path
/sas/sasfoundation. A programmer
doesn't always add Memsize or a directory path /sas/sasfoundation in his
code.

Therefore the output in his log file will not have Memsize or a directory
path
/sas/sasfoundation.

In all the log files, there is an assignment variable named Real Time with
a numeric value.

Real Time value is normally low. The value range between 0.0 - 0.9
Real Time value is high if the value is 1.0

It's hard to replicate the sas log files. So I created sample data in each
text file. The most important in the sas log files are the parameters and its
values which are Memsize= with a numeric value, a path directory
/sas/sasfoundation and Real Time :
with a numeric value

I have several problems with my problems: I hope you can help me.

1) the If else statement I am getting an error message. My syntax must be incorrect because the entire statement is throwing an error.

Code:

For example in filew.log if these items don't exist Memsize, SASFoundation and also if Real Time row value is less than 1.0 than no data is output to progflag.cvs.txt

Code:

Results for Example: output - progflag,cvs.txt:
Memsize        Second      SASEXE                filename
    200                          SASFoundation        file1x.log.txt
    100                          SASFoundation        file2x.log.txt
    400         5.1                                         filez.log.txt

2) I am not getting any data in progflag.cvs.txt even though Memsize, and SASFoundation are in some of the log files that the program reads in

3) I am not receiving the progflag.cvs.txt attachment via email

Code:

4) I added *.log | awk  because I want the program to read in log files with the .log extension only. There are other files in the directory that have different extensions. I am getting an error message.

Code:

5) The program searches for 3 selected items in each log files.

Code:

Item 1# :  Memsize.  Memsize statement stores numeric values.  For example memsize=400. the program output the column name (memsize) and its value and the filename to progflag.csv.txt

Code:

example - output -   progflag.csv.txt:
memsize   Second    SASEXE       filename
 200                                        file1x.log.txt

Code:

Item 2# :  Real Time:    For example, variable Real Time stores a row value. Example Real Time:  0.1.

In my program  the variable Real Time alias name is  Second. If Second < 1.0 then output no data

Code:

For example in filez.log, Real Time: 0.1. IF SECOND  < 1.0 then no numeric value output to the column named Second in progflag.csv.txt

Code:

example - output -  progflag.csv.txt:
memsize   Second      SASEXE                        filename
100                         SASFoundation                file2x.log.txt

Code:

Item6# : another situation, if For example,  the row value for Real Time is 5.1 then output the numeric value to column name Second in  progflag.csv.txt.

Code:

IF SECOND  > 1.0 then output the value to the column in progflag.csv.txt

Code:

example - output - progflag.csv.txt:

memsize  Second      SASEXE       filename
 400         5.1                            filez.log.txt

Code:

item7#:   if the program finds  the directory path /SASFoundation (SASEXE) then output SASFoundation to column name sasexe in  progflag.csv.txt

Code:

cd /tmp/logs
*.log | awk -F '[=:;.]' '
  function pr() {if(NR>1) printf "%s\t%s\t%s\t%s\n", K[1],K[2],K[3],K[0]}
  BEGIN {
      printf "MEMSIZE\tSECOND\tSASEXE\tFilename\n"
      for(i=split("memsize ,Real Time ,SASFoundation",A,",");i;i--) L[A[i]]=i
  }
  FNR==1 {
      pr()
       K[0]=FILENAME
      K[1]=K[2]=K[3]=x
  }
  $1 in L {v=$2;gsub("^[/ ]*","",v);gsub(/ *$/,"",v);K[L[$1]]=v}
  END{pr(
{if ($1 || $2>1.0 || $ 3 &&  $0) printf $1 "\t" $2 "\t"" $3"\t" $0"\t; elseif($2>1.0 else print ''}'
' *.log > progflag.csv

[ -s progflag.csv ] && mailx -s "subject text -a "Programs flagged" receiver@domain.com < progflag.csv

file2x.log.txt (438 Bytes)

filew.log.txt (342 Bytes)

filez.log.txt (439 Bytes)

file1x.log.txt (454 Bytes)

progflag.csv.txt (285 Bytes)