Visit Our UNIX and Linux User Community


HELP: Shell Script to read a Log file line by line and extract Info based on KEYWORDS matching


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting HELP: Shell Script to read a Log file line by line and extract Info based on KEYWORDS matching
# 1  
Old 07-24-2012
Network HELP: Shell Script to read a Log file line by line and extract Info based on KEYWORDS matching

I have a LOG file which looks like this
Code:
 
Import started at: Mon Jul 23 02:13:01 EDT 2012
Initialization completed in 2.146 seconds.
--------------------------------------------------------------------------------
--
Import summary for Import item: PolicyInformation
--------------------------------------------------------------------------------
--
Batch size: 1.
Commit count: 100.
Error Tolerance Level: 1.
Error Count: 0.
Amount of data processed: 10.
Amount of objects committed: 8.
Data Importer initialization time: 1 seconds.
Data Importer execution began: Mon Jul 23 02:13:03 EDT 2012
Data Importer execution ended: Mon Jul 23 02:13:06 EDT 2012
Data Importer completed in 2.735 seconds.
Total flush time: 0 seconds.
Total commit time: 0.008 seconds.
Affected tables (4):
Table name: PolicyHdr, Affected number of rows: 118.
Table name: PolicyDetl, Affected number of rows: 76.
Table name: ClaimSeq, Affected number of rows: 211.
Table name: Claimant, Affected number of rows: 116.
----------------------------------------------------------------------------------
 
Program exiting with exit code: 0.
Import completed successfully with no errors.
Import ended at: Mon Jul 23 02:13:06 EDT 2012
Import completed in 4.891 seconds.

I have KEYWORDS whcih are these
Error Count:
Amount of data processed:
Import completed in
and many more


Whats the best way to open and loop thru the file line by line looking for Keywords like these and extracting rest of the information .

i've looked/read several posts but nothing quiet what i need.
i tried GREP/AWK & SED but nothing is giving me what i want... (obviously im not an expert scriptor)

now this is what i've cooked up so far
Code:
#!/bin/bash
file="/home/myuser/importlog.txt"
val1='Import completed in'
while IFS= read -r line
do
let count++ # display $line or do somthing with $line
#echo "$count $line"
fit=`expr substr "$line" 1 17`
if [ $fit == $val1 ]
then
echo "Found IT $line"
fi
done <"$file"

it fails with line 9: [: too many arguments

not sure why its failing when matching the $fit == $val1 coz at command line it works

$ line='Import completed in'
$ fit=`expr substr "$line" 1 17`
$ echo $fit
Import completed


can any of you experts quickly cut my agony by helping me out... sincerely appreciated....
# 2  
Old 07-24-2012
Start with this:

Write all of your keywords or phrases to a file, call it key.txt.

Code:
grep -Ff key.txt your_logfile_name

Beware of captialization and spaces, grep will do an EXACT match on your keywords.
And print the whole line.
# 3  
Old 07-24-2012
Thanks Jim,
but i want the control inside the loop so i can do other operations like Substring on FOUND Lines
OR
look for additional info based on what keyword it found....

so i was looking something like this..

case Keyword1
{
do something
}
case Keyword2
{
do onething
do twothing
}
# 4  
Old 07-24-2012
For case, you can always use case...

Code:
while read STRING
do
case "$STRING" in
*abc*)
        stuff
        ;;

*qwerty*)
        other stuff
        ;;

abcde)
        yet more stuff
        ;;
esac
done < inputfile

This User Gave Thanks to Corona688 For This Post:
# 5  
Old 07-24-2012
i tried 2nd CASE statement

Code:
#!/bin/bash
file="/home/mbuser/log.txt"
val1='Load completed in'
fit=holy
err=crap
while IFS= read -r line
do
        let count++ # display $line or do somthing with $line
        #echo "$count $line"
case "$line" in
'*Import completed*')
        echo "Completed"
        fit=`expr substr "$line" 21 7`
        ;;
'*Error Count:*')
        echo "ERRROR"
        err=`expr substr "$line" 13 7`
        ;;
esac
done <"$file"
echo $fit $err

it outputs
holy crap ==< which i have initialized at top....as if it didnt go in the case statement.. what am i missing?
Moderator's Comments:
Mod Comment Code tags for code, please.

Last edited by Corona688; 07-24-2012 at 06:20 PM..
# 6  
Old 07-24-2012
I don't think those strings in the ) need to be in single quotes.
# 7  
Old 07-24-2012
if i dont put single QUOTES it fails with

./krun: line 11: syntax error near unexpected token `Count:*'
./krun: line 11: `*Error Count:*)'

Double Quotes doesnt make any difference.... same behaviou as Single Quote

Previous Thread | Next Thread
Test Your Knowledge in Computers #209
Difficulty: Medium
The development work for OSPF prior to its publication as open standard was undertaken largely by the Digital Equipment Corporation.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script UNIX to read text file line by line

i have a text file as belows, it includes 2 columns, 1st is the column name, 2nd is the file_name data_file.txt column_name file_name col1 file1 col2 file2 col3 file1 col4 file1 col5 file2 now, i would like to... (4 Replies)
Discussion started by: tester111
4 Replies

2. Shell Programming and Scripting

Replace line in file with line in another file based on matching string

HI Can any one guide me how to achieve this task. I have 2 files env.txt #Configuration.Properties values identity_server_url = http://identity.test-hit.com:9783/identity/service/user/register randon_password_length = 6 attachment_file_path = /pass/temp/attachments/... (1 Reply)
Discussion started by: nikilbr86
1 Replies

3. Shell Programming and Scripting

Replace and add line in file with line in another file based on matching string

Hi, I want to achieve something similar to what described in another post: The difference is I want to add the line if the pattern is not found. File 1: A123, valueA, valueB B234, valueA, valueB C345, valueA, valueB D456, valueA, valueB E567, valueA, valueB F678, valueA, valueB ... (11 Replies)
Discussion started by: jyu3
11 Replies

4. Shell Programming and Scripting

Shell script to read multiple options from file, line by line

Hi all I have spent half a day trying to create a shell script which reads a configuration file on a line by line basis. The idea of the file is that each will contain server information, such as IP address and various port numbers. The line could also be blank (The file is user created). Here... (1 Reply)
Discussion started by: haggismn
1 Replies

5. Shell Programming and Scripting

Replace line in file with line in another file based on matching string

Hi I am not the best scripter in the world and have run into a issue which you might be able to guide me on... I have two files. File1 : A123, valueA, valueB B234, valueA, valueB C345, valueA, valueB D456, valueA, valueB E567, valueA, valueB F678, valueA, valueB File2: C345,... (5 Replies)
Discussion started by: luckycharm
5 Replies

6. Shell Programming and Scripting

Shell script to read a text file line by line & process it...

Hi , I am trying to write an shell, which reads a text file (from a location) having a list of numbers of strictly 5 digits only ex: 33144 Now my script will check : 1) that each entry is only 5 digits & numeric only, no alphabets, & its not empty. 2)then it executes a shell script called... (8 Replies)
Discussion started by: new_to_shell
8 Replies

7. Shell Programming and Scripting

get the fifth line of a text file into a shell script and trim the line to extract a WORD

FOLKS , i have a text file that is generated automatically of an another korn shell script, i want to bring in the fifth line of the text file in to my korn shell script and look for a particular word in the line . Can you all share some thoughts on this one. thanks... Venu (3 Replies)
Discussion started by: venu
3 Replies

8. Shell Programming and Scripting

how can u read a file line by line in shell script ?

hello , plz help for below script req:- how can we read a file line by line in shell script ? (4 Replies)
Discussion started by: abhigrkist
4 Replies

9. UNIX for Dummies Questions & Answers

shell script to read file line by line

Hi, I need to read a text file from shell script line by line and copy the feilds of each line. Below is the complete requirement. I've text file which contains ... pgm1 file11 file12 file13 pgm2 file21 file22 pgm3 file31 file32 file33 I'll give input as... (4 Replies)
Discussion started by: ani12345
4 Replies

10. Shell Programming and Scripting

shell script to read a line in gps receiver log file and append that line to new file

Hi, I have gps receiver log..its giving readings .like below Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GPSD,R=1 $GPGSV,3,1,11,08,16,328,40,11,36,127,00,28,33,283,39,20,11,165,00*71... (3 Replies)
Discussion started by: gudivada213
3 Replies

Featured Tech Videos