Shell Programming and Scripting

Hi guys, I'm hoping you can help me here. I've knocked up a script that looks at a (huge) log file, and pulls from each line the hour of each transaction and how long each transaction took.

The data is stored sequentially as:

07:01 blah blah blah 12456 blah
07:03 blah blah blah 234 blah
08:02 blah blah blah 9 blah

My script works, but because it searches through the whole script for each occurence of hour x , it is searching the file 24 times and taking about 45 seconds to complete.

The pseudo code for what I have written is:

HOUR=0
while HOUR < 24; do
read in file line by line
find line matching $HOUR and pass column 5 to array
HOUR = HOUR + 1
done

Obviously my actual code is a lot more complex than that, I can add it if it helps, but I think the above simplifies my request for help

Does anyone have any idea how to speed up the way I am doing this? I had a go at doing it like:

for variable in `cat ANYOLDFILE`
do
HOUR = first 2 chars of line
TRANSACTIONTIME=column 4
done

But that seemed to take just as long and I'm sure there must be someway to speed this up.

Using ksh by the way.

Thanks in advance for any help.

to calculate the totals per hour:

nawk -f dl.awk ANYOLDFILE

dl.awk:

Code:

{
  tot[substr($1,1,2)] += $5
}
END {
  for(i in tot)
    printf("Hour->[%s]: [%d]\n", i, tot[i])
}

Hi, thanks for that, really handy script, but I was probably a bit vague in my request (my heads a bit fried from getting it working - albeit too slowly)

The output I require is along the lines of:


Hour Count Trans time <1 sec Count Trans time <2 sec etc.
07-08___________54______________________02
08-09___________23______________________07
09-10___________00 _____________________04
...
23-24___________14______________________25

So what it displays is total number of transactions taking less than x seconds between 7 and 8 am and between 8 and 9 etc.

I'm bit lost of what the meaning of the input columns is AND what you're trying to report on.
Can you try to elaborate on the meaning of the input columns and the desired output. You're quoting a desired output, but it does not jive with your sample output and the description of the output is somewhat hard to decipher.

Another sample input and a corresponding output would help along with the more detailed explanation.

Ok, I'll try and explain it a little better.

The input file contains details of transactions including time of transaction and total time taken. The desired output is the total number of transactions per time taken, broken down into hourly periods.

An example of the input would be:

07:01 blah blah blah 3 blah
07:03 blah blah blah 2 blah
08:02 blah blah blah 1 blah
08:05 blah blah blah 1 blah
09:10 blah blah blah 3 blah

And the desired output would be:

Time___________1 second_______2 second________3 second
07:00 - 07:59_______0____________1______________1
08:00 - 08:59_______2____________0______________0
09:00 - 09:59_______0____________0______________1

Which shows that between 7 and 8 there was one transaction that took 2 seconds and one transaction that took 3 seconds. Between 8 and 9 there were two transactions that took 2 seconds and between 9 and 10 there was 1 transaction that took 3 seconds.

Hope that clarfies it a bit.