Extract Log lines with Thread-(regex)


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract Log lines with Thread-(regex)
# 1  
Old 08-22-2011
Extract Log lines with Thread-(regex)

Hi everyone,

Fist of all I must confess that I am pretty new in the Unix environment and especially to shell scripting, however due to work related requirements I have started to analyze software specific logs.

The logs are structured so that it records by sessionID AND/OR Thread number, the problem is that the two do not necessarily are in all lines related to a session.

What we currently do is cat the log file and pipe it to a perl script that would print to screen all lines with the Session ID, from there we would take each line with Thread-(3 to 8 numbers) and repeat the same entry adding each unrepeated Thread-number to the entry and > to a text file.

I was wondering if you would know a way to create a qualifier i.e."-allthreads" in Perl or bash that would directly capture each occurence of Thread-/\d{3-8}/ and add it to the search.

Ideas? I dont know if I was clear,
Thanks in advance.

Sushimatt.
# 2  
Old 08-22-2011
Quote:
Originally Posted by sushimatt
...
What we currently do is cat the log file and pipe it to a perl script that would print to screen all lines with the Session ID, from there we would take each line with Thread-(3 to 8 numbers) and repeat the same entry adding each unrepeated Thread-number to the entry and > to a text file.
...
A much better way to describe this is to provide a short, real-life demonstration using dummy data. Cook up some data in a text file and show us the input to and output from the Perl script. And the changes you want further.

As for the question about passing a qualifier - you could pass a single-character switch to the perl interpreter and then use the "getopts" method of the "GetOpt::Std" module to
(a) parse the switch and (b) perform relevant actions based on the switch.

Link to the CPAN page:

getopts - search.cpan.org

tyler_durden
# 3  
Old 09-29-2011
My example.

Well, The thing is exactly that, I dont get it to return any data. The files have a structure where each line have a thread number and data after it:

2011-09-01 00:05:00,286 9220793 [Thread-17608] DATA DATA DATA.

I simply want the script to return in a line each instance of thread numbers through piping a cat command.

Code:
#!/usr/bin/perl
#2011-09-01 00:05:00,286 9220793 [Thread-17608]
my ($thread) = $_ =~ /\[Thread-\s*(.+?)\]/;
%seen = ();
@threads = ();

while (<STDIN>) {
        if ($_ =~ m/\[Thread-\s*(.+?)\]/) { unless ($seen{$thread}) {
        push (@threads, $thread);
        $seen{$thread}++;
        }}  else {
        $seen{$thread}++;
        }
}
        foreach (@threads) {
        print "$_";
        }

Any ideas?
# 4  
Old 09-29-2011
Could this help you ?
use this
Code:
if ($_ =~ m/\[Thread-\s*$thread\]/)

instead of
Code:
if ($_ =~ m/\[Thread-\s*(.+?)\]/)

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need grep regex to extract multiline text between two strings

I have a file conatining the below: --- 10.9.16.116: /tmp/5835113081224811756.jar: hash: e6df90d38fa86f0e289f73d79cd2cfd2a29954eb /tmp/4603745991442278706.jar: hash: e6df90d38fa86f0e289f73d79cd2cfd2a29954eb 10.9.14.126: /tmp/conf/extra/httpd-ssl.conf: hash:... (1 Reply)
Discussion started by: mohtashims
1 Replies

2. Shell Programming and Scripting

Perl to extract whole number or decimal in regex

In the perl below I am trying to extract and print specic values from patterns using multiple regex. One of the patterns AF= may be a whole number or a decimal but I can not seem to capture both. I think it is the regex .*AF=(\d+\.\d+); as it is expecting a #.#### and it may only be a #. I tried... (2 Replies)
Discussion started by: cmccabe
2 Replies

3. Shell Programming and Scripting

ksh sed - Extract specific lines with mulitple occurance of interesting lines

Data file example I look for primary and * to isolate the interesting slot number. slot=`sed '/^primary$/,/\*/!d' filename | tail -1 | sed s'/*//' | awk '{print $1" "$2}'` Now I want to get the Touch line for only the associate slot number, in this case, because the asterisk... (2 Replies)
Discussion started by: popeye
2 Replies

4. Shell Programming and Scripting

How to get next 3 lines after regex?

HI, Just wanted to get some advice on how i can get the next 3 lines after a regex: Say i have a file which contains something like below: client467 master tcp ether tor0141 4467 query tcp ether tor0141 4467 client468 master tcp ether tor2141 4468 ... (17 Replies)
Discussion started by: reignangel2003
17 Replies

5. Shell Programming and Scripting

Can you extract (remove) lines from log files?

I use "MineOS" (a linux distro with python scripts and web ui included for managing a Minecraft Server). The author of the scripts is currently having a problem with the Minecraft server log file being spammed with certain entries. He's working on clearing up the spam. But in the meantime, I'm... (8 Replies)
Discussion started by: nbsparks
8 Replies

6. Shell Programming and Scripting

Regex: Extract substring between 2 separator

Hi Input: aa-bb-cc-dd.ee.ff.gg Output: dd I want to get the word after the last '-' until the first dot I have tried with regex lookbehind and lookahead like this: (?<=-).*(?=\.) but his returns too much bb-cc-dd.ee.ff (7 Replies)
Discussion started by: chitech
7 Replies

7. Shell Programming and Scripting

Search for a pattern,extract value(s) from next line, extract lines having those extracted value(s)

I have hundreds of files to process. In each file I need to look for a pattern then extract value(s) from next line and then search for value(s) selected from point (2) in the same file at a specific position. HEADER ELECTRON TRANSPORT 18-MAR-98 1A7V TITLE CYTOCHROME... (7 Replies)
Discussion started by: AshwaniSharma09
7 Replies

8. UNIX for Dummies Questions & Answers

Extract lines with specific words with addition 2 lines before and after

Dear all, Greetings. I would like to ask for your help to extract lines with specific words in addition 2 lines before and after these lines by using awk or sed. For example, the input file is: 1 ak1 abc1.0 1 ak2 abc1.0 1 ak3 abc1.0 1 ak4 abc1.0 1 ak5 abc1.1 1 ak6 abc1.1 1 ak7... (7 Replies)
Discussion started by: Amanda Low
7 Replies

9. Shell Programming and Scripting

SED With Regex to extract Email Address

Hi Folks, In my program, I have a variable which consists of multiple lines. i need to use each line as an input. My intention is to extract the email address of the user in each line and use it to process further. The email address could be anywhere in the whole line. But there will be only... (5 Replies)
Discussion started by: ragz_82
5 Replies

10. Shell Programming and Scripting

Perl REGEX - How do extract a string in a line?

Hi Guys, In the following line: cn=portal.090710.191533.428571000,cn=groups,dc=mp,dc=rj,dc=gov,dc=br I need to extract this string: portal.090710.191533.428571000 As you can see this string always will be bettween "cn=" and "," strings. Someone know one regular expression to... (4 Replies)
Discussion started by: maverick-ski
4 Replies
Login or Register to Ask a Question