Log file text parsing


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Log file text parsing
# 1  
Old 12-28-2009
Log file text parsing

I'm new to scripting and was wondering if there was a way to accomplish what I want below using shell script(s).

If there is a log file as follows, where the id is the unique id of a process, with the timestamp of when the process began and completed displayed, would it be possible to find the time taken for each process (i.e. end time - start time)? Also, would it be possible to parse the text log to see if there are any processes that never completed (i.e. those that do not have a end id #X posted)?

BEGIN ID #1 - 2009-12-27 13:31:40.123894
END ID #1 - 2009-12-27 13:31:40.127205
BEGIN ID #2 - 2009-12-27 13:31:42.445600
BEGIN ID #3 - 2009-12-27 13:31:42.445635
END ID #3 - 2009-12-27 13:31:42.451483
END ID #2 - 2009-12-27 13:31:42.578831


I'm not looking for the actual script. Just want to know if something like this is possible, and if so, what/where should I be looking at?

Thanks in advance to any insight anyone can provide.
# 2  
Old 12-28-2009
Tools think about awk to accomplish

Since you are only asking for ideas...

Code:
awk on the file
# read thru the file
   val_id = id # extracted from string

# check to see if a BEGIN record
   if (BEGIN)
      Arr_Begin(valid_id) = extracted time
# an array to keep track of # of mentions
      Arr_Hits(valid_id) = Arr_Hits(valid_id) + 1

# check to see if an END record
   if (END) 
      Arr_End(valid_id) = extracted time
# an array to keep track of # of mentions
      Arr_Hits(valid_id) = Arr_Hits(valid_id) + 1

# now go thru the valid_id array for your conditions
# look for recs where Arr_Hits =1
# etc...

# 3  
Old 12-28-2009
Hello ,
Code:
#!/usr/local/bin/perl -w
use strict;
open(FILE,"<your_log_file"); #open the file
while(<FILE>) #read file
{
if(/BEGIN/){push(@list,$_);}
elsif(/END/)
{
my @rec=split(/\s/,$_);
foreach $val (@list){my @org_list=split(/\s/,$val);if($rec[2]==$org_list[2]) 
{
my @time1=split(/:/,$rec[5]);
my @time2=split(/:/,$org_list[5]);
print $rec[2],"process executed for",$time2[0]-$time1[0],":",$time2[1]-$time1[1],":",$time2[2]-$time1[2];
}else print $rec[2],"is a daemon";
} #foreach 
}#while

The algorithm goes like this,
if the tag is begin, push it in some list,
if the tag is end, run a loop through the list and check for its BEGIN. if found then find the difference in time otherwise, say that that the process was a daemon(never ended)
Regards,

P.S. sorry for the messy code. in a hurry..
# 4  
Old 12-28-2009
Thanks guys for your suggestions. I now have an idea on how to approach this, so thanks again.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Text file parsing and comparison

I have two files (first.txt and second.txt): more first.txt cat mammal lizard reptile Elephant mammal ant Insecta more second.txt ant termite ant army_ant (9 Replies)
Discussion started by: cs_novice
9 Replies

2. Shell Programming and Scripting

Parsing text file

Hi Friends, I am back for the second round today - :D My input text file is this way Home friends friendship meter Tools Mirrors Downloads My Data About Us Help My own results BLAT Search Results ACTIONS QUERY SCORE START END QSIZE IDENTITY CHRO STRAND ... (7 Replies)
Discussion started by: jacobs.smith
7 Replies

3. Shell Programming and Scripting

Parsing text file

I'm totally stumped with how to handle this huge text file I'm trying to deal with. I really need some help! Here is what is looks like: ab1ba67c331a3d731396322fad8dd71a3b627f89359827697645c806091c40b9 0.2 812a3c3684310045f1cb3157bf5eebc4379804e98c82b56f3944564e7bf5dab5 0.6 0.6... (3 Replies)
Discussion started by: comp8765
3 Replies

4. Programming

Parsing a Text file using C++

I was trying to parse the text file, which will looks like this ###XYZABC#### ############ int = 4 char = 1 float = 1 . . ############ like this my text file will contains lots of entries and I need to store these entries in the map eg. map.first = int and map.second = 4 same way I... (5 Replies)
Discussion started by: agupta2
5 Replies

5. Shell Programming and Scripting

Need help parsing a text file

I have a text file: router1#sh ip blah blah | incl --- Gi2/8 10.60.4.181 --- 10.60.123.175 11 0000 0000 355K Gi2/8 10.60.83.28 --- 224.10.10.26 11 F9FF 3840 154K Gi2/8 10.60.83.198 --- ... (1 Reply)
Discussion started by: streetfighter2
1 Replies

6. UNIX for Dummies Questions & Answers

Help parsing and replacing text with file name

Hi everyone, I'm having trouble figuring this one out. I have ~100 *.fa files with multiple lines of fasta sequences like this: file1.fa >xyzsequence atcatgcacac...... ataccgagagg..... atataccagag..... >abcsequence atgagatatat..... acacacggd..... atcgaacac.... agttccagat.... The... (2 Replies)
Discussion started by: mycoguy
2 Replies

7. Shell Programming and Scripting

Parsing text from file

Any ideas? 1)loop through text file 2)extract everything between SOL and EOL 3)output files, for example: 123.txt and 124.txt for the file below So far I have: sed -n "/SOL/,/EOL/{p;/EOL/q;}" file Here is an example of my text file. SOL-123.go something goes here something goes... (0 Replies)
Discussion started by: ndnkyd
0 Replies

8. Shell Programming and Scripting

Need help in parsing text file contents

Hi, I need some help in extracting the Exception block between the lines 21 Feb 01:18:54:146 ERROR com.orbits.frameworks.integrationframework.ValidationException - Caught exception in validateRequest() (PID=565584) and 21 Feb 01:18:55:149 INFO ... (0 Replies)
Discussion started by: Alecs
0 Replies

9. Shell Programming and Scripting

Text File Parsing

Hey Guys.I am a newbie on Bash Shell Scripting and Perl.And I have a question about file parsing. I have a log file which contains reports about a communication device.I need to take some of the reports from the log file.Its hard to explain the issue.but shortly I can say that, the reports has a... (2 Replies)
Discussion started by: Djlethal
2 Replies

10. Shell Programming and Scripting

Parsing Lines from a text file

Hello Guru's I know there are people who live to be able to help people like me. That's why I'm here! I have a text file that has a corresponding address information that I need to be able to get maybe as a Unix Function. The text file contains the following: AIX17_JB_C;... (3 Replies)
Discussion started by: sallender
3 Replies
Login or Register to Ask a Question