Read in Multiple log files and output selected variables and values to cvs file


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Read in Multiple log files and output selected variables and values to cvs file
# 1  
Old 06-02-2016
Read in Multiple log files and output selected variables and values to cvs file

Quote:
1. The OS is AIX
2. I am using bash shell
3. The format of text files is based on generated sas programs that produce

sas log files. The sas programmers sometimes add the following
parameters to their code Memsize, and a directory path
/sas/sasfoundation. A programmer
doesn't always add Memsize or a directory path /sas/sasfoundation in his
code.

Therefore the output in his log file will not have Memsize or a directory
path
/sas/sasfoundation.

In all the log files, there is an assignment variable named Real Time with
a numeric value.

Real Time value is normally low. The value range between 0.0 - 0.9
Real Time value is high if the value is 1.0

It's hard to replicate the sas log files. So I created sample data in each
text file. The most important in the sas log files are the parameters and its
values which are Memsize= with a numeric value, a path directory
/sas/sasfoundation and Real Time :
with a numeric value
I have several problems with my problems: I hope you can help me.

1) the If else statement I am getting an error message. My syntax must be incorrect because the entire statement is throwing an error.

Code:
For example in filew.log if these items don't exist Memsize, SASFoundation and also if Real Time row value is less than 1.0 than no data is output to progflag.cvs.txt

Code:
Results for Example: output - progflag,cvs.txt:
Memsize        Second      SASEXE                filename
    200                          SASFoundation        file1x.log.txt
    100                          SASFoundation        file2x.log.txt
    400         5.1                                         filez.log.txt

2) I am not getting any data in progflag.cvs.txt even though Memsize, and SASFoundation are in some of the log files that the program reads in


3) I am not receiving the progflag.cvs.txt attachment via email

Code:
4) I added *.log | awk  because I want the program to read in log files with the .log extension only. There are other files in the directory that have different extensions. I am getting an error message.

Code:
5) The program searches for 3 selected items in each log files.

Code:
Item 1# :  Memsize.  Memsize statement stores numeric values.  For example memsize=400. the program output the column name (memsize) and its value and the filename to progflag.csv.txt

Code:
example - output -   progflag.csv.txt:
memsize   Second    SASEXE       filename
 200                                        file1x.log.txt

Code:
Item 2# :  Real Time:    For example, variable Real Time stores a row value. Example Real Time:  0.1.

In my program  the variable Real Time alias name is  Second. If Second < 1.0 then output no data

Code:
For example in filez.log, Real Time: 0.1. IF SECOND  < 1.0 then no numeric value output to the column named Second in progflag.csv.txt

Code:
example - output -  progflag.csv.txt:
memsize   Second      SASEXE                        filename
100                         SASFoundation                file2x.log.txt

Code:
Item6# : another situation, if For example,  the row value for Real Time is 5.1 then output the numeric value to column name Second in  progflag.csv.txt.

Code:
IF SECOND  > 1.0 then output the value to the column in progflag.csv.txt

Code:
example - output - progflag.csv.txt:

memsize  Second      SASEXE       filename
 400         5.1                            filez.log.txt

Code:
item7#:   if the program finds  the directory path /SASFoundation (SASEXE) then output SASFoundation to column name sasexe in  progflag.csv.txt

Code:
cd /tmp/logs
*.log | awk -F '[=:;.]' '
  function pr() {if(NR>1) printf "%s\t%s\t%s\t%s\n", K[1],K[2],K[3],K[0]}
  BEGIN {
      printf "MEMSIZE\tSECOND\tSASEXE\tFilename\n"
      for(i=split("memsize ,Real Time ,SASFoundation",A,",");i;i--) L[A[i]]=i
  }
  FNR==1 {
      pr()
       K[0]=FILENAME
      K[1]=K[2]=K[3]=x
  }
  $1 in L {v=$2;gsub("^[/ ]*","",v);gsub(/ *$/,"",v);K[L[$1]]=v}
  END{pr(
{if ($1 || $2>1.0 || $ 3 &&  $0) printf $1 "\t" $2 "\t"" $3"\t" $0"\t; elseif($2>1.0 else print ''}'
' *.log > progflag.csv

[ -s progflag.csv ] && mailx -s "subject text -a "Programs flagged" receiver@domain.com < progflag.csv

 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk GSUB read field values from multiple text files

My program run without error. The problem I am having. The program isn't outputting field values with the column headers to file.txt. Each of the column headers in file.txt has no data. MEMSIZE SECOND SASFoundation Filename The output results in file.txt should show: ... (1 Reply)
Discussion started by: dellanicholson
1 Replies

2. Shell Programming and Scripting

How to read each line from input file, assign variables, and echo to output file?

I've got a file that looks like this (spaces before first entries intentional): 12345650-000005000GL140227 ANNUAL HELC FEE EN 22345650-000005000GL140227 ANNUAL HELC FEE EN 32345650-000005000GL140227 ANNUAL HELC FEE EN I want to read through the file line by line,... (6 Replies)
Discussion started by: Scottie1954
6 Replies

3. Shell Programming and Scripting

Read record from the text file contain multiple separated values & assign those values to variables

I have a file containing multiple values, some of them are pipe separated which are to be read as separate values and some of them are single value all are these need to store in variables. I need to read this file which is an input to my script Config.txt file name, first path, second... (7 Replies)
Discussion started by: ketanraut
7 Replies

4. Shell Programming and Scripting

Read record from the text file & assign those values to variables in the script

For eg: I have sample.txt file with 4 rows of record like: user1|password1 user2|password2 user3|password3 user4|password4 The username and password is sepsrated by '|' I want to get the 1st row value from the file and assign it to two different variables(username and password) in my... (1 Reply)
Discussion started by: priya001
1 Replies

5. Shell Programming and Scripting

Read file and for each line replace two variables, add strings and save output in another file

Hi All, I have a file, let's call it "info.tmp" that contains data like this .. ABC123456 PCX333445 BCD789833 I need to read "info.tmp" and for each line add strings in a way that the final output is put /logs/ua/dummy.trigger 'AAA00001.FTP.XXX.BLA03A01.xxxxxx(+1)' where XXX... (5 Replies)
Discussion started by: Andy_ARG
5 Replies

6. Shell Programming and Scripting

Read 1-line file and separate into multiple variables

I have one line files with 17 records separated by a semi-colon. I need to create a variable from each record, which I can do via a separate awk for each one, but I know there has to be a better way. Along with pulling out the variable, I need to convert some url coding like a + to a space, etc.... (4 Replies)
Discussion started by: numele
4 Replies

7. Shell Programming and Scripting

Read variables and their values from file

Hi, I want to read the variables and the values from the txt file and compare these values with the ones computed by script. for ex: say var.txt contains the variable names and their values: one 1 two 2 three 3 The value of variables "one" "two" and "three" will be computed in the script... (3 Replies)
Discussion started by: bhushana
3 Replies

8. Shell Programming and Scripting

Read multiple log files and create output file and put the result

OS : Linux 2.6.9-67 - Red Hat Enterprise Linux ES release 4 Looking for a script that reads the following log files that gets generated everynight between 2 - 5am Master_App_20090717.log Master_App1_20090717.log Master_App2_20090717.log Master_App3_20090717.log... (2 Replies)
Discussion started by: aavam
2 Replies

9. UNIX for Dummies Questions & Answers

trouble using read to store values in variables from command output

I know there are caveats about using read in pipelines because read is treated by a subshell. I know this but I can't think of any way to accomplish this regardless, I'm still a rookie. I hope somebody will be able to interpret what it is that I'm trying to accomplish and correct me. ... (2 Replies)
Discussion started by: ProGrammar
2 Replies

10. Shell Programming and Scripting

how can i read text file and assign its values to variables using shell

Hello, I have a cat.dat file, i would like shell to read each 3 lines and set this 3 lines to 3 different variables. my cat.dat is: 11 12 +380486461001 12 13 +380486461002 13 14 +380486461003 i want shell to make a loop and assign 1st line to student_id, 2nd line to... (4 Replies)
Discussion started by: rosalinda
4 Replies
Login or Register to Ask a Question