Count number of unique patterns from a log file


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Count number of unique patterns from a log file
# 1  
Old 07-10-2011
Count number of unique patterns from a log file

Hello Everyone

I need your help in fixing this issue., I have a log file which has data of users logging in to an application.

I want to search for a particular pattern in the log

Code:
ISSessionValidated=N

If this key word is found , the above 8 lines will contain the name of the user who's session did not validate

for example .. the log section would be some thing like this

Code:
12:34:45 CT Session 334d3dx3z322cx User : appleInc2232
transfering tag..
transfering session..
transfering data..
level one tag completed ..
file created...
logging into main application
user session transfer created..
ISSessionValidated=N
Exception caught.. Session invalidated ..
Exiting thread 334d3dx3z322cx

I need to search the log file which has multiple sessions like this and check if the session validated = N if the keyword is found then i should check the above 8 lines and see if the user is appleInc2232 if found a count should be incremented in a file. There are over 11k sessions like this in a single log file.But the pattern is like this.

Operating system : SOLARIS 10
Single Log file size above : 4 GB
Over 700 log files with the same data is availble ( if you give a start as to how to proceed it will be helpful )
log files pattern is something like this

Code:
binded.session.01_01_2011
binded.session.01_02_2011
binded.session.01_03_2011
binded.session.01_04_2011.gz
binded.session.01_05_2011.gz
binded.session.01_06_2011.gz
---- More ---

Any help would be greatly appreciated. Thanks.

Last edited by Scott; 07-10-2011 at 06:47 AM.. Reason: Code tags
# 2  
Old 07-10-2011
You can start with this:
Code:
perl -0072 -ne '/appleInc2232.*ISSessionValidated=N/s && $i++;END{print "$i\n"}' logfile

This User Gave Thanks to bartus11 For This Post:
# 3  
Old 07-10-2011
Error Need Shell script

Perl is not running in our machine can you please suggest a shell script ?
# 4  
Old 07-10-2011
Try:
Code:
nawk '/appleInc2232.*ISSessionValidated=N/{i++}END{print i}' RS=":" logfile

This User Gave Thanks to bartus11 For This Post:
# 5  
Old 07-10-2011
nawk nawk...
Code:
[n]awk '{FS=/appleInc2232.*ISSessionValidated=N/}END{print NF-1}' file

This User Gave Thanks to shamrock For This Post:
# 6  
Old 07-10-2011
Thanks both of you, it works,. i have one last question how can i make this work with compressed log files .gz files. ?
# 7  
Old 07-10-2011
Code:
gzcat logfile.gz | nawk '/appleInc2232.*ISSessionValidated=N/{i++}END{print i}' RS=":"

This User Gave Thanks to bartus11 For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Count number of unique values in each column of array

What is an efficient way of counting the number of unique values in a 400 column by 1000 row array and outputting the counts per column, assuming the unique values in the array are: A, B, C, D In other words the output should look like: Value COL1 COL2 COL3 A 50 51 52... (16 Replies)
Discussion started by: Geneanalyst
16 Replies

2. Shell Programming and Scripting

Find and count unique date values in a file based on position

Hello, I need some sort of way to extract every date contained in a file, and count how many of those dates there are. Here are the specifics: The date format I'm looking for is mm/dd/yyyy I only need to look after line 45 in the file (that's where the data begins) The columns of... (2 Replies)
Discussion started by: ronan1219
2 Replies

3. Shell Programming and Scripting

Awk to Count Multiple patterns in a huge file

Hi, I have a file that is 430K lines long. It has records like below |site1|MAP |site2|MAP |site1|MODAL |site2|MAP |site2|MODAL |site2|LINK |site1|LINK My task is to count the number of time MAP, MODAL, LINK occurs for a single site and write new records like below to a new file ... (5 Replies)
Discussion started by: reach.sree@gmai
5 Replies

4. Shell Programming and Scripting

count the number of occurring patterns in a file.

Hi, I have a file with a '|' pipe delimeter. I want to find number of counts for a particular pattern in particular field. Is it possible to do it in a single command? 1) want to find total number of "0" in field 4. 2) want to find total number of different records in field 4 ( similar to... (5 Replies)
Discussion started by: rudoraj
5 Replies

5. Shell Programming and Scripting

List unique values and count instances in .csv file

I need to take the second column of a .csv file and count the number of instances of each unique value in that same second column. I'd like the output to be value,count sorted by most instances. Thanks for any guidance! Data example: 317476,317756,0 816063,318861,0 313123,319091,0... (4 Replies)
Discussion started by: batcho
4 Replies

6. Shell Programming and Scripting

Unique count from flat file

Hello Guys I have a flat file with '|~|' delimited When I use to record count using below command awk -FS"+" ' {print $colno}' filename | wc -l the count is fine But when I am trying to find the unique number of record the o/p is always 1 awk -FS"+" ' {print $colno}'... (11 Replies)
Discussion started by: Pratik4891
11 Replies

7. Shell Programming and Scripting

How to count Unique Values from a file.

Hi I have the following info in a file - <Cell id="25D"/> <Cell id="26A"/> <Cell id="26B"/> <Cell id="26C"/> <Cell id="27A"/> <Cell id="27B"/> <Cell id="27C"/> <Cell id="28A"/> I would like to know how would you go about counting all... (4 Replies)
Discussion started by: Prega
4 Replies

8. Shell Programming and Scripting

Count number of Nodes created and write it to a Log file

Dear Experts, I have to count the number of AddressRecords formed in bbc.xml file using unix script file. For example: for below pasted file, I need to write an output to a log file as "No. of Address Records Created=4". Snippet of bbc.xml:- <?xml version="1.0" encoding="UTF-8" ?> -... (1 Reply)
Discussion started by: phani333
1 Replies

9. Shell Programming and Scripting

Count lines between two patterns inside a file

Hi, Im doing a script to find the number of lines included inside a file newly. These lines are in between #ifdef FLAG1 and #else or #endif or #else and #endif. I tried like this, awk '/#ifdef Flag1/,/#e/{print}' aa.c | wc -l awk '/#ifndef Flag1/,/#endif/{print}' aa.c | awk... (6 Replies)
Discussion started by: priyadarshini
6 Replies

10. Shell Programming and Scripting

Script to count unique number of user loged in

Hi all, I am taking my first course in unix and having some problems. I would line to create a scrip that does the following: 1) will tell me the number of users logged in. If a user is logged more than once just count it only ones. 2) want to be able to display the number of users that... (1 Reply)
Discussion started by: elchalateco
1 Replies
Login or Register to Ask a Question