Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Find keywords in multiple log files Post 302974513 by dellanicholson on Tuesday 31st of May 2016 09:59:17 AM
Old 05-31-2016
Find keywords in multiple log files

The Problem that I am having is when the code ran and populated the progflag.csv file, columns MEMSIZE, SECOND and SASEXE were blank. The next problems are the IF else statement isn't working and the email function isn't sending the progflag.csv attachment.

a. What I want the program to do is to only read in all files that have a
.log. extension. There other files in the same directory with different extension such as .html, and .ksh,

There are four steps that occur when the program output data to progflag.csv (data comes from multiple log files that are read in by the program):

1 for example if the following criteria in file12.log.20163005 doesn't exist, such as MEMSIZE , a directory path will the text /SASFoundation and the variable Real Time row value(Real Time 0.1) is less than 1.0, no data is outputted to progflag.csv .

2. if in file9.log.20163005, the following variables and values exist for example "MEMSIZE=200" and variable Real Time (Real Time 4.0) row value is More than 1.0, output the values and the column names to the proflag.csv.

3. if in file25.log.20163005, for example there is a directory path will text /SASFoundation and the variable Real Time (Real Time 0.2) row value is less than 1.0, output the values and its column name to the proflag.csv

An example in the progflag.csv results from the steps are as follows:

MEMSIZE SECOND SASEXE filename:
200 4.0 file1.log.20163005
SASFoundation file25.log.20163005

4. last, step only send an email if progflag.csv file rows are populated.

Here is the code:

Code:
cd /tmp/*.log.*
awk -F '[=:;.]' '
  function pr() {if(NR>1) printf "%s\t%s\t%s\t%s\n", K[1],K[2],K[3],K[0]}
  BEGIN {
      printf "MEMSIZE\tSECOND\tSASEXE\tFilename\n"
      for(i=split("memsize ,Real Time ,SASFoundation",A,",");i;i--) L[A[i]]=i
  }
  FNR==1 {
      pr()
       K[0]=FILENAME
      K[1]=K[2]=K[3]=x
  }
  $1 in L {v=$2;gsub("^[/ ]*","",v);gsub(/ *$/,"",v);K[L[$1]]=v}
  END{pr()}
if MEMSIZE OR SECOND >1.0 OR  SASEXE AND Filename then
' *.log.* > progflag.csv

[ -s progflag.csv ] && mailx -s "subject text -a "Programs flagged" receiver@domain.com < progflag.csv
ELSE ''

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to find particular string in multiple files

Hello friends, I have find a paticular string from the files present in my user for example: a username and password is hardcoded in multiple files which present in the my user.so I have to search about username in which files it is available.there are several dirctories are there,so... (5 Replies)
Discussion started by: sivaranga001
5 Replies

2. Shell Programming and Scripting

Finding 50k Keywords in 3k files

Hi, I have a file with about 50k keywords. I have a requirement to scan about 3k files to identify which filename has which keyword i.e. an output like following: File1,Keyword1 File1,Keyword2 File3,Keyword1 ..... I have written a shell script which takes each of the 3k files, searches... (4 Replies)
Discussion started by: rjains
4 Replies

3. Shell Programming and Scripting

Script to find & replace a multiple lines string across multiple php files and subdirectories

Hey guys. I know pratically 0 about Linux, so could anyone please give me instructions on how to accomplish this ? The distro is RedHat 4.1.2 and i need to find and replace a multiple lines string in several php files across subdirectories. So lets say im at root/dir1/dir2/ , when i execute... (12 Replies)
Discussion started by: spfc_dmt
12 Replies

4. UNIX for Dummies Questions & Answers

finding keywords in many files using grep

Hi to all Sorry for the confusion because I did not explain the task clearly. There are many .hhr files in a folder There are so many lines in these .hhr files but I want only the following 2 lines to be transferred to the output file. The keyword No 1 and all the words in the next line They... (5 Replies)
Discussion started by: raghulrajan
5 Replies

5. UNIX for Advanced & Expert Users

Need to search for keywords within files modified at a certain time

I have a huge list of files in an Unix directory (around 10000 files). I need to be able to search for a certain keyword only within files that are modified between certain date and time, say for e.g 2012-08-20 12:30 to 2012-08-20 12:40 Can someone let me know what would be the fastest way... (10 Replies)
Discussion started by: virtual123
10 Replies

6. Shell Programming and Scripting

Find keywords, and append at the end of line

Task: Find keywords in each line, and append at the end of line; if not found in the line, do nothing. the code is wrong. how to make it work. thanks a lot. cat keywords.txt | while read line; do awk -F"|" '{if (/$line/) {print $0"$line , ";} else print;}' outfile.txt > tmp ... (9 Replies)
Discussion started by: dtdt
9 Replies

7. Shell Programming and Scripting

Search files in directory for keywords using bash

I have ~100 text files in a directory that I am trying to parse and output to a new file. I am looking for the words chr,start,stop,ref,alt in each of the files. Those fields should appear somewhere in those files. The first two fields of each new set of rows is also printed. Since this is on a... (7 Replies)
Discussion started by: cmccabe
7 Replies

8. Shell Programming and Scripting

Find keywords in multiple log files

I have several problems with my program: I hope you can help me. 1) the If else statement isn't working . The IF Else syntax is: If MEMSIZE OR sasfoundation (SASEXE) OR Real Time(second) >1.0 and Filename, output column name and value to csv or else nothing Example progflag,cvs:... (13 Replies)
Discussion started by: dellanicholson
13 Replies

9. Shell Programming and Scripting

Grep multiple keywords from a file

I have a script that will search for a keyword in all the log files. It work just fine. LOG_FILES={ "/Sandbox/logs/*" } for file in ${LOG_FILES}; do grep $1 $file done This only works for 1 keyword. What if I want to search for more then 1 keywords, say 4 or maybe even... (10 Replies)
Discussion started by: Loc
10 Replies

10. UNIX for Beginners Questions & Answers

Find and replace from multiple files

Hello everybody, I need your help. I have a php site that was expoited, the hacker has injected into many php files a phishing code that was discovered and removed in order to have again a clean code. Now we need to remove from many php files that malware. I need to create a script that find and... (2 Replies)
Discussion started by: ninocap
2 Replies
CSV(3pm)						User Contributed Perl Documentation						  CSV(3pm)

NAME
Class::CSV - Class based CSV parser/writer SYNOPSIS
use Class::CSV; my $csv = Class::CSV->parse( filename => 'test.csv', fields => [qw/item qty sub_total/] ); foreach my $line (@{$csv->lines()}) { $line->sub_total('$'. sprintf("%0.2f", $line->sub_total())); print 'Item: '. $line->item(). " ". 'Qty: '. $line->qty(). " ". 'SubTotal: '. $line->sub_total(). " "; } my $cvs_as_string = $csv->string(); $csv->print(); my $csv = Class::CSV->new( fields => [qw/userid username/], line_separator => " "; ); $csv->add_line([2063, 'testuser']); $csv->add_line({ userid => 2064, username => 'testuser2' }); DESCRIPTION
This module can be used to create objects from CSV files, or to create CSV files from objects. Text::CSV_XS is used for parsing and creating CSV file lines, so any limitations in Text::CSV_XS will of course be inherant in this module. EXPORT None by default. METHOD
CONSTRUCTOR parse the parse constructor takes a hash as its paramater, the various options that can be in this hash are detailed below. Required Options o fields - an array ref containing the list of field names to use for each row. there are some reserved words that cannot be used as field names, there is no checking done for this at the moment but it is something to be aware of. the reserved field names are as follows: "string", "set", "get". also field names cannot contain whitespace or any characters that would not be allowed in a method name. Source Options (only one of these is needed) o filename - the path of the CSV file to be opened and parsed. o filehandle - the file handle of the CSV file to be parsed. o objects - an array ref of objects (e.g. Class::DBI objects). for this to work properly the field names provided in fields needs to correspond to the field names of the objects in the array ref. o classdbi_objects - depreciated use objects instead - using classdbi_objects will still work but its advisable to update your code. Optional Options o line_separator - the line seperator to be included at the end of every line. defaulting to " " (unix carriage return). new the new constructor takes a hash as its paramater, the same options detailed in parse apply to new however no Source Options can be used. this constructor creates a blank CSV object of which lines can be added via add_line. ACCESSING lines returns an array ref containing objects of each CSV line (made via Class::Accessor). the field names given upon construction are available as accessors and can be set or get. for more information please see the notes below or the perldoc for Class::Accessor. the lines accessor is also able to be updated/retrieved in the same way as individual lines fields (examples below). Example retrieving the lines: my @lines = @{$csv->lines()}; removing the first line: pop @lines; $csv->lines(@lines); sorting the lines: @lines = sort { $a->userid() <=> $b->userid() } @lines: $csv->lines(@lines); sorting the lines (all-in-one way): $csv->lines([ sort { $a->userid() <=> $b->userid() } @{$csv->lines()} ]); Retrieving a fields value there is two ways to retrieve a fields value (as documented in Class::Accessor). firstly you can call the field name on the object and secondly you can call "get" on the object with the field name as the argument (multiple field names can be specified to retrieve an array of values). examples are below. my $value = $line->test(); OR my $value = $line->get('test'); OR my @values = $line->get(qw/test test2 test3/); Setting a fields value setting a fields value is simmilar to getting a fields value. there are two ways to set a fields value (as documented in Class::Accessor). firstly you can simply call the field name on the object with the value as the argument or secondly you can call "set" on the object with a hash of fields and their values to set (this isn't standard in Class::Accessor, i have overloaded the "set" method to allow this). examples are below. $line->test('123'); OR $line->set( test => '123' ); OR $line->set( test => '123', test2 => '456' ); Retrieving a line as a string to retrieve a line as a string simply call "string" on the object. my $string = $line->string(); new_line returns a new line object, this can be useful for to "splice" a line into lines (see example below). you can pass the values of the line as an ARRAY ref or a HASH ref. Example my $line = $csv->new_line({ userid => 123, domainname => 'splicey.com' }); my @lines = $csv->lines(); splice(@lines, 1, 0, $line); OR splice(@{$csv->lines()}, 1, 0, $csv->new_line({ userid => 123, domainname => 'splicey.com' })); add_line adds a line to the lines stack. this is mainly useful when the new constructor is used but can of course be used with any constructor. it will add a new line to the end of the lines stack. you can pass the values of the line as an ARRAY ref or a HASH ref. examples of how to use this are below. Example $csv->add_line(['house', 100000, 4]); $csv->add_line({ item => 'house', cost => 100000, bedrooms => 4 }); OUTPUT string returns the object as a string (CSV file format). print calls "print" on string (prints the CSV to STDOUT). SEE ALSO
Text::CSV_XS, Class::Accessor AUTHOR
David Radunz, <david@boxen.net> COPYRIGHT AND LICENSE
Copyright 2004 by David Radunz This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.10.0 2007-02-08 CSV(3pm)
All times are GMT -4. The time now is 05:06 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy