Unix/Linux Go Back    


UNIX for Dummies Questions & Answers This forum is closed for new posts. Please post beginner questions to learn unix and learn linux in this forum UNIX for Beginners Questions & Answers

Find keywords in multiple log files

UNIX for Dummies Questions & Answers


 
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 05-31-2016
dellanicholson dellanicholson is offline
Registered User
 
Join Date: Jul 2015
Last Activity: 26 September 2017, 11:33 PM EDT
Posts: 46
Thanks: 20
Thanked 0 Times in 0 Posts
Find keywords in multiple log files

The Problem that I am having is when the code ran and populated the progflag.csv file, columns MEMSIZE, SECOND and SASEXE were blank. The next problems are the IF else statement isn't working and the email function isn't sending the progflag.csv attachment.

a. What I want the program to do is to only read in all files that have a
.log. extension. There other files in the same directory with different extension such as .html, and .ksh,

There are four steps that occur when the program output data to progflag.csv (data comes from multiple log files that are read in by the program):

1 for example if the following criteria in file12.log.20163005 doesn't exist, such as MEMSIZE , a directory path will the text /SASFoundation and the variable Real Time row value(Real Time 0.1) is less than 1.0, no data is outputted to progflag.csv .

2. if in file9.log.20163005, the following variables and values exist for example "MEMSIZE=200" and variable Real Time (Real Time 4.0) row value is More than 1.0, output the values and the column names to the proflag.csv.

3. if in file25.log.20163005, for example there is a directory path will text /SASFoundation and the variable Real Time (Real Time 0.2) row value is less than 1.0, output the values and its column name to the proflag.csv

An example in the progflag.csv results from the steps are as follows:

MEMSIZE SECOND SASEXE filename:
200 4.0 file1.log.20163005
SASFoundation file25.log.20163005

4. last, step only send an email if progflag.csv file rows are populated.

Here is the code:


Code:
cd /tmp/*.log.*
awk -F '[=:;.]' '
  function pr() {if(NR>1) printf "%s\t%s\t%s\t%s\n", K[1],K[2],K[3],K[0]}
  BEGIN {
      printf "MEMSIZE\tSECOND\tSASEXE\tFilename\n"
      for(i=split("memsize ,Real Time ,SASFoundation",A,",");i;i--) L[A[i]]=i
  }
  FNR==1 {
      pr()
       K[0]=FILENAME
      K[1]=K[2]=K[3]=x
  }
  $1 in L {v=$2;gsub("^[/ ]*","",v);gsub(/ *$/,"",v);K[L[$1]]=v}
  END{pr()}
if MEMSIZE OR SECOND >1.0 OR  SASEXE AND Filename then
' *.log.* > progflag.csv

[ -s progflag.csv ] && mailx -s "subject text -a "Programs flagged" receiver@domain.com < progflag.csv
ELSE ''

Attached Files
File Type: txt progflag.csv.txt (287 Bytes, 2 views)
Sponsored Links
    #2  
Old Unix and Linux 05-31-2016
vbe's Unix or Linux Image
vbe vbe is offline Forum Staff  
Moderator
 
Join Date: Sep 2005
Last Activity: 18 October 2017, 10:16 AM EDT
Location: Switzerland - GE
Posts: 6,470
Thanks: 288
Thanked 551 Times in 513 Posts
Your code:

Code:
if MEMSIZE OR SECOND >1.0 OR  SASEXE AND Filename then
' *.log.* > progflag.csv

[ -s progflag.csv ] && mailx -s "subject text -a "Programs flagged" receiver@domain.com < progflag.csv
ELSE ''

Should not run as I see syntax errors... I am surprised you got anything as output...
I cant guess the errors ro why its not working as the code isnt complet nor do we know what shell you are using...

Code:
if MEMSIZE OR SECOND >1.0 OR  SASEXE AND Filename then

this is notthe correct way to write an if condition statement nor is

Code:
ELSE ''

and I suspect this is just part of issue...
Sponsored Links
    #3  
Old Unix and Linux 05-31-2016
dellanicholson dellanicholson is offline
Registered User
 
Join Date: Jul 2015
Last Activity: 26 September 2017, 11:33 PM EDT
Posts: 46
Thanks: 20
Thanked 0 Times in 0 Posts
Read in multiple log files and Output selected keywords and its value to a cvs file


Code:
I have several problems with my program:  I hope you can help me.


Code:
1) the If else statement isn't working I am getting a syntax error.

 .  The IF Else syntax is:
     If MEMSIZE OR sasfoundation (SASEXE) OR Real Time(second) >1.0 and Filename, output column name and value to csv or else nothing


Code:
Example: output - progflag,cvs:
Memsize                 Second           SASEXE             filename                        
    400                       4.0             SASFoundaion     file11.log.20120314


Code:
2) I am not getting  any data in the csv file.


Code:
3) I am not getting the attachment in the email. I am not receiving the cvs file attachment via email


Code:
My program read in multiple files with .log, extension. For example file12.log.20120314. The program search for 3 selected items in each log files. 2 items  Memsize and SASFoundation may not be  in all log files. But Real Time : and its value are in every log files.


Code:
Item 1# input read in the program from log file:  Memsize .   Memsize statement stores numeric values.  For example memsize=400. the program output the column name (memsize) and its value and the filename to a csv file


Code:
example output - progflag.csv:
memsize              filename
 400                     file12.log.20120314


Code:
Item 2# the input read into the program from log files:  
Real Time:  row value.


Code:
For example, input read from log files, 
The row value for Real Time is 4.0.  Real Time :  4.0.

In my program Real Time is named Second.
[/CODE]

Code:
For example -input, SECOND stores  4.0.  IF SECOND  > 1.0 then output the column name and its value to a cvs file


Code:
example - progflag.csv:

Second                 filename
 4.0                      file11.log.20120314

If Real Time  row value is less than 1.0 then output no data to the cvs file.

Example Real Time: 0.2 0.2 is less than 1.0

Code:
item3#:   if the program find  the directory path /SASFoundation (SASEXE) then output  the directory path to a cvs file

Example progflag.cvs
Second SASEXE filename
4.0 SASFoundaion file11.log.2012031
[/CODE]

Code:
cd /tmp/*.log.*
awk -F '[=:;.]' '
  function pr() {if(NR>1) printf "%s\t%s\t%s\t%s\n", K[1],K[2],K[3],K[0]}
  BEGIN {
      printf "MEMSIZE\tSECOND\tSASEXE\tFilename\n"
      for(i=split("memsize ,Real Time ,SASFoundation",A,",");i;i--) L[A[i]]=i
  }
  FNR==1 {
      pr()
       K[0]=FILENAME
      K[1]=K[2]=K[3]=x
  }
  $1 in L {v=$2;gsub("^[/ ]*","",v);gsub(/ *$/,"",v);K[L[$1]]=v}
  END{pr(
{if ($1 || $2>1.0 || $ 3 &&  $0) printf $1 "\t" $2 "\t"" $3"\t" $0"\t; elseif($2>1.0 else print ''}'
' *.log.* > progflag.csv

[ -s progflag.csv ] && mailx -s "subject text -a "Programs flagged" receiver@domain.com < progflag.csv


Last edited by dellanicholson; 06-01-2016 at 05:46 PM.. Reason: ADD [CODE] [/CODE]
Sponsored Links
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Find keywords in multiple log files dellanicholson Shell Programming and Scripting 13 06-07-2016 09:52 PM
Find keywords, and append at the end of line dtdt Shell Programming and Scripting 9 04-23-2014 11:56 AM
finding keywords in many files using grep raghulrajan UNIX for Dummies Questions & Answers 5 03-14-2012 11:05 AM
Script to find & replace a multiple lines string across multiple php files and subdirectories spfc_dmt Shell Programming and Scripting 12 03-07-2012 09:05 AM
Finding 50k Keywords in 3k files rjains Shell Programming and Scripting 4 05-10-2011 01:02 PM



All times are GMT -4. The time now is 10:47 AM.