Calculate avg response time on hourly basis


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Calculate avg response time on hourly basis
# 1  
Old 06-05-2013
Calculate avg response time on hourly basis

Hi,

I am trying to calculate avg response time on hourly basis from the log file which has millions of records.

As of now I am trying with creating temp file which will have lines with unique id and start time and end time and after that another script will run on this temp file to calcualte avg response time.
Reason I posted this ques in forum becuase this way takes more than an hour to create temp file.

Is there any way we can do it quicker?
Note:These UNIQID is not coming in sequence.
Code:
2012-06-04 13:04:19,324 UNIQID1
2012-06-04 13:04:20,120 UNIQID1
2012-06-04 13:05:19,324 UNIQID2
2012-06-04 13:06:20,120 UNIQID2
2012-06-04 13:07:19,324 UNIQID3
2012-06-04 13:08:20,120 UNIQID3
2012-06-04 13:08:49,324 UNIQID4
2012-06-04 13:09:50,120 UNIQID4


Last edited by Franklin52; 06-05-2013 at 07:55 AM.. Reason: Please use code tags
# 2  
Old 06-05-2013
Whats your current script which is taking hours of time? May be we can try to finetune the same for you Smilie
# 3  
Old 06-05-2013
Code:
for uid in ${uids}; do
count=`grep "$uid" test.log|wc -l`
if [ "${count}" -ne "0" ]; then
unique_uids[counter]="$uid"
#echo "${unique_uids[counter]}"
let counter=counter+1
fi
done
echo ${unique_uids[@]}
echo $counter
echo " Unique No:" ${#unique_uids[@]}
echo uid StartTime EndTime" > $log
for unique_uids in ${unique_uids[@]} ; do
 
responseTime=`cat $i|grep "${unique_uids}" |awk '{split($2,Arr,":|,"); print Arr[1]*3600000+Arr[2]*60000+Arr[3]*1000+Arr[4]}'|sort -n`
echo $unique_uids $responseTime >> $log
done

Output will be like this
Code:
UID Start time end time
UNIQID1 13:04:19,324 13:04:20,120

---------- Post updated at 07:12 AM ---------- Previous update was at 05:33 AM ----------

pls let me know if something in script is not clear. The first for loop is find unique no of ids and then grepping those uniq ids to find start and end time.

Thanks for you time!!

Last edited by Franklin52; 06-05-2013 at 07:55 AM.. Reason: Please use code tags
# 4  
Old 06-05-2013
This
Code:
responseTime=`cat $i|grep "${unique_uids}" |awk '{split($2,Arr,":|,"); print Arr[1]*3600000+Arr[2]*60000+Arr[3]*1000+Arr[4]}'|sort -n`

may be some rearranged
Code:
responseTime=$(awk '$0~u {split($1,Arr,":|,"); print 1000*(Arr[1]*3600+Arr[2]*60+Arr[3])+Arr[4]}' u="${unique_uids}" $i | sort -n)

# 5  
Old 06-05-2013
Thanks for looking into this,

This exactly work like my previous code, how ever the pain area to get avg response time for each hour.

like in 24 hr format 00,01,02...23 for each hour avg responsetime would be assume if 10 hist then total response time( end time - start time) sum of response times for 10 hits/ 10

Thanks,
# 6  
Old 06-07-2013
Hi Jotne, Can you pls explain me what this common does. inside awk $0~u
# 7  
Old 06-07-2013
Quote:
Originally Posted by random_thoughts
Hi Jotne, Can you pls explain me what this common does. inside awk $0~u
True if the content of $0 match the content of variable u
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to count respon time max min avg for nginx logs?

Hi All, need your help, i want count respon time max and average my nginx logs, based on hourly or minutes per api... my nginx.log sample : 10.1.1.1 - - "POST /v2/api/find/outlet/ HTTP/1.1" 200 2667 "-" "okhttp/3.12.0" "118.215.153.47" 0.178 0.178 . 10.1.1.1 - - "POST... (4 Replies)
Discussion started by: fajar_3t3
4 Replies

2. Shell Programming and Scripting

Calculate Time diff in milli milliseconds(Time format : HH:MM:SS,NNN)

Hi All, I have one file which contains time for request and response. I want to calculate time difference in milliseconds for each line. This file can contain 10K lines. Sample file with 4 lines. for first line. Request Time: 15:23:45,255 Response Time: 15:23:45,258 Time diff... (6 Replies)
Discussion started by: Raza Ali
6 Replies

3. Shell Programming and Scripting

How to calculate avg values of csv file using shell scripting .?

hi all i have a reporting work and i want it to be automated using shell scripting kindly let me know how can i make that possibe . eg data are :... (2 Replies)
Discussion started by: Avinash shaw
2 Replies

4. Shell Programming and Scripting

Crontab on hourly basis

Hi.. I need to run the script on hourly basis. How do I write the crontab on hourly basis i.e, 9:00, 10:00.....22:00.. 23:00 hours Please let me know if the below is correct one for crontab on hourly basis. 00 * * * * ksh myscript.ksh > /dev/null Regards, John (3 Replies)
Discussion started by: scriptscript
3 Replies

5. Shell Programming and Scripting

FTP a file on Hourly basis

Hi, I have to upload a file test_201105281100.txt to a ftp location. The files will be created on hourly basis like test_201105281100.txt, test_201105281200.txt & so on. After a file is uploaded successfully, I need to rename the file as test_201105281100.success & if it is not uploaded... (11 Replies)
Discussion started by: SunilB2011
11 Replies

6. Shell Programming and Scripting

Calculate response time from log

I have a running log file (jboss_server.log) which rotates at midnight . I need to constantly check and calculate the time for each thread and alert if it doesnt complete within 60 minute. For example my log file has following printed . I want to run a script in cron every 30 minutes and... (2 Replies)
Discussion started by: gubbu
2 Replies

7. Shell Programming and Scripting

Calculate age of a file | calculate time difference

Hello, I'm trying to create a shell script (#!/bin/sh) which should tell me the age of a file in minutes... I have a process, which delivers me all 15 minutes a new file and I want to have a monitoring script, which sends me an email, if the present file is older than 20 minutes. To do... (10 Replies)
Discussion started by: worm
10 Replies

8. Shell Programming and Scripting

how to grep string from hourly basis files

dear all, pls help on this script.. i have many files which will be created every mins in particular directory. i want to grep a particular string from only for unique hour files. from the below code i want to grep a string from only 9th hour files . Ex files: -rw-r--r-- 1 root ... (5 Replies)
Discussion started by: steve2216
5 Replies

9. Solaris

Monitoring the output of 'top' command on hourly basis.

I need to capture the following data on an hourly basis through cronjob scheduling:- 1. load averages 2. Total no. of processes. 3. CPU state 4. Memory 5. Top 3 process details. All the above information is available through the command 'top'. But here we need to automate the same and... (4 Replies)
Discussion started by: subharai
4 Replies
Login or Register to Ask a Question