Count occurances of X Y Z in a file in 1 go.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Count occurances of X Y Z in a file in 1 go.
# 1  
Old 03-02-2009
Count occurances of X Y Z in a file in 1 go.

Hi. I need to count multiple occurrences of X Y Z in a file in 1 go. At the moment I have the following scripts:
ssh readonly@$ServerIP 'YEAR=xx;DAY=xx;MONTH=xx;LMONTH=xx;for i in 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 \
16 17 18 19 20 21 22 23; do cat /var/SP/log/cre/access.log_$YEAR$MONTH$DAY*_$i | grep -c "HTTP/1.1\" \"503";done'>>$sshEF
this goes for HTTP1.1 503... then there is 500, 400, 403 and 404 which runs the same thing..

Now I have to look for HTTP response codes in the hourly log files on an apache web server.. and count them.
At the moment my grep command runs through the log files once for each response type.

I have also tried
var_500=0;var_503=0;var_400=0;var_403=0;var_404=0
for i in 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23;
do cat /var/SP/log/cre/access.log_$YEAR$MONTH$DAY*_$i | while read line
do
variable=`echo $line | awk '{print$7}'` #print response code with awk
case $variable in #ERRORS 1
500 ) var_500=`expr $var_500 + 1`;;
503 ) var_503=`expr $var_503 + 1`;;
400 ) var_400=`expr $var_400 + 1`;;
403 ) var_403=`expr $var_403 + 1`;;
404 ) var_404=`expr $var_404 + 1`;;
* ) hello=hello
esac
datetime=$i
Server=`hostname`
date="$YEAR/$MONTH/$DAY"
for p in 500 503 400 403 404;
do
err_desc="HTTP/1.1 $p"
Value=`echo $(var_$p)`
echo "vl$Servername,$date,$err_desc,$Value"
done
done

But the CPU utilization and time is too long to be a viable solution. ( 4min CPU time at over 3% utilization, whereas the grep -c is 10 seconds @ 0.1%)

Is there an easy way to count multiple things in a file in one go? Or should I just stick with grep -c?


Thanks in advance
# 2  
Old 03-02-2009
This look like a typical problem for awk. I mean 100% awk. Can you post sample input file and desired output?
# 3  
Old 03-02-2009
This is sample input:
10.113.98.16 10.113.155.52 - - [02/Mar/2009:09:00:01 +0200] "GET /mi_icons/misc/png240/divider.png HTTP/1.1" 200 227 "http://live.vo
dafone.com/!1GjhKhab-irZoHNZX3giab" "Mozilla/5.0 (SymbianOS/9.3; U; Series60/3.2 Samsung/I8510/XXHJ3; Profile/MIDP-2.1 Configuration
/CLDC-1.1 ) AppleWebKit/413 (KHTML, like Gecko) Safari/413 UP.Link/6.3.1.12.0" cookie:"$Version=1;JSESSIONID=947BCC46C9C2A6DF3460044
72E043C9E" "-" from:"41.26.19.165" 3g:"no" "27768552299" D:"1275" trusted:"-" x-up-from:"41.26.19.165"
10.113.98.16 10.113.155.52 - - [02/Mar/2009:09:00:01 +0200] "GET /img/ca/logo-red-bg.gif HTTP/1.1" 200 429 "http://owafe011.vodacomm
i.co.za/join.ravenriley.com/track/picdom;6279:RR:RR,0,0,0,/" "SonyEricssonW880i/R8BA Browser/NetFront/3.3 Profile/MIDP-2.0 Configura
tion/CLDC-1.1 UP.Link/6.3.1.12.0" cookie:"-" "-" from:"41.28.172.226" 3g:"no" "27825900717" D:"1129" trusted:"-" x-up-from:"41.28.17
2.226"

I highlighted the response codes.

Desired output will be something like:
HTTP/1.1" 500,<count value>
HTTP/1.1" 503,<count value>
HTTP/1.1" 400,<count value>
HTTP/1.1" 403,<count value>
HTTP/1.1" 404,<count value>
# 4  
Old 03-02-2009
Just to put you on track, try this:

Code:
#! /bin/bash

awk '
{total[$9] += 1} 
END {
	for (i in total) 
	print i, total[i]
}' /var/log/apache2/access.log /var/log/apache2/access.log.1

edit: If this doesn't produce the desired output try $10 instead of $9 as it seems that the HTTP response code in your sample file is on position 10

Last edited by ripat; 03-02-2009 at 03:35 AM..
# 5  
Old 03-02-2009
Thanks Smilie I will have a look
# 6  
Old 03-02-2009
Superb!! Thanks ripat!
I goes to about 3.5% CPU Utilization, but finishes in about 15 seconds, which is fair game for me.

Output:
502 3
304 1359
503 88
404 467
200 31817
500 301
302 207

*Edit: But if there is another way to do it which uses very little CPU, please feel free to post a reply

Last edited by msullivan; 03-02-2009 at 04:03 AM..
# 7  
Old 03-02-2009
I changed the awk command to:
awk '{total[$10] += 1} END{for (i in total) print "HTTP/1.1 "i, total[i]}'
and output is:
HTTP/1.1 304 591
HTTP/1.1 503 24
HTTP/1.1 404 402
HTTP/1.1 200 21480
HTTP/1.1 500 5
HTTP/1.1 302 141

Is there a way I can pass a variable to the awk command so that the output will be:
var=00
00 HTTP/1.1 304 xxx
var=01
01 HTTP/1.1 304 xxx

*Edit: I changed the command to nawk and the result is a huge drop in CPU utilization, and quicker results.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Grep pattern file and count occurances

Guys I am trying grep to read from pattern file and count occurances of each word. input is firstplace secondplace secondpot patternfile is place first second i want the following. 1-count number of times keywords in patternfile occurs in begining of each line in input file. so... (12 Replies)
Discussion started by: ahfze
12 Replies

2. Shell Programming and Scripting

Count the number of occurances for multiple files

I have some text files as shown below. I would like to add the values of each string. Your help would be appreciated!! file1.txt SUS 2 PRS 2 ALI 1 PRS 1 GLK 2 file2.txt PRS 3 GLK 6 SUS 18 Desired output SUS 20 PRS 6 (2 Replies)
Discussion started by: arch
2 Replies

3. Shell Programming and Scripting

Count the occurances of numbers

I have a file as shown below. I need to count the unique occurrences of numbers in the first and second column only if the third column is <= 10. Otherwise print the corresponding numbers as zero. Thanks in advance!:) 58 80 40.74 76 80 9.78 76 80 8 12 6 9 30 28 8.23 45 12 ... (13 Replies)
Discussion started by: andreaalex
13 Replies

4. Shell Programming and Scripting

Get number of occurances in a file

I want to get the count of occurance of a user in a file. abc abc bcd bcd bcd xyz The result should be like abc 2 bcd 3 xyz 1 (2 Replies)
Discussion started by: Sanjeev Yadav
2 Replies

5. Shell Programming and Scripting

How to count number of occurances of string in a file?

Gurus, Need little guidance. I have A.txt and B.txt file. B.txt file contains Unique strings. Sample content of B.txt file for which i cut the fourth column uniquely and output directed to B.txt file And A.txt file contains the above string as a fourth column which is last column. So A.txt... (7 Replies)
Discussion started by: Shirisha
7 Replies

6. Shell Programming and Scripting

nawk-how count the number of occurances of a pattern, when don't know the pattern

I've written a script to count the total size of SAN storage LUNs, and also display the LUN sizes. From server to server, the LUNs sizes differ. What I want to do is count the occurances as they occur and change. These are the LUN sizes: 49.95 49.95 49.95 49.95 49.95 49.95 49.95 49.95... (2 Replies)
Discussion started by: cyber111
2 Replies

7. UNIX for Dummies Questions & Answers

Count occurances in a file

Hi, I have a fixed width file in the following format Sr.No A.No Name 1 2 PPP 3 4 PPP 5 6 TTT 7 8 OOO 9 10 OOO 11 12 OOO The 3rd column starts at position 10 and ends at 15. I want to count the number of occurances of each Name and output to a file Example in the... (2 Replies)
Discussion started by: samit_9999
2 Replies

8. Shell Programming and Scripting

Count occurances of a character in a file

I want to find the number of occurences of a character in a file. How do i do it. Eg: $cat file1.txt Welcome to World of Unix. $ If i want to find the occurences of 'o' then I should be getting 3. Thanks. (6 Replies)
Discussion started by: Shivdatta
6 Replies

9. Shell Programming and Scripting

count the number chracters occurances in a line

Hi Could anybody tell me how to count the number of occurances of a character within a LINE. actually i have a single line with words seperated with '+' chracter e.g. abc+def+ghj+jkl+asd i want to separte the words above. Please provide the necessary logic in the form of a SHELL... (6 Replies)
Discussion started by: skyineyes
6 Replies

10. Shell Programming and Scripting

Find occurances of a directory and count

I am writing my first shell script to do the following: Find all occurrences of a directory from the pwd Delete the directory (which is a hidden directory) Provide feedback w/ the number of directories deleted The problems I am having are two-fold: The user may not have the appropriate... (2 Replies)
Discussion started by: stringman
2 Replies
Login or Register to Ask a Question