Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Script (ksh) to get data in every 30 mins interval for the given date

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script (ksh) to get data in every 30 mins interval for the given date
# 8  
Old 10-04-2013
Quote:
Originally Posted by rpm120
Great thanks for your explanation.Smilie

I was trying out this piece of code below
Code:
awk -v rd="2013/03/07" -v st="00:00:00" -v et="00:29:59" '$1==rd && $2>=st && $2<=et' log | grep -c "successful"

Because i wanted make that script very generic such that in which ever place the pattern "successful" is found it should get the count. Since I want to use the same script for two more diffrent log in which the successful message can be in any place not neccesarily in the last ($NF == "successful").

But the problem i was facing here was i was not able to increment the timestamp values i.e "st" and "et" with 30 minutes and get a logic for that to repeat the piece of code for every 30 minutes.

So can you please suggest me a solution for that.
Assuming you still want the same output format, I would just change the:
Code:
$1 == dd && $NF == "successful" { s[$2 + 0, $3 > 29]++ }

in the script I gave you to:
Code:
$1 == dd && /successful/ { s[$2 + 0, $3 > 29]++ }

Adding the boilerplate around the piece of code you suggested to produce the same output (calling awk 48 times calling grep 48 times and reading your log file 49 times) instead of calling awk once to read your log file once seems extremely wasteful.

If you really want to process your log file 48 times instead of once, I can come up with a way to do that, but I'll need some explanation as to why doing that would be better than the trivial change to my script that is shown above.
This User Gave Thanks to Don Cragun For This Post:
rpm120
# 9  
Old 10-04-2013
Lightbulb need to get the sum of the successful batches at the last
Quote:
Originally Posted by Don Cragun
Assuming you still want the same output format, I would just change the:
Code:
$1 == dd && $NF == "successful" { s[$2 + 0, $3 > 29]++ }

in the script I gave you to:
Code:
$1 == dd && /successful/ { s[$2 + 0, $3 > 29]++ }

Adding the boilerplate around the piece of code you suggested to produce the same output (calling awk 48 times calling grep 48 times and reading your log file 49 times) instead of calling awk once to read your log file once seems extremely wasteful.

If you really want to process your log file 48 times instead of once, I can come up with a way to do that, but I'll need some explanation as to why doing that would be better than the trivial change to my script that is shown above.

Thanks again...Smilie

I do not have any specific reason to go with that solution (awk -v rd="2013/03/07" -v st="00:00:00" -v et="00:29:59" '$1==rd && $2>=st && $2<=et' log | grep -c "successful"), I just wanted to let you know that i was trying out in that method. Thanks for the feedback on the same as well.

Its good for me to go with your solution.Smilie

But I need one more functionality to be added with this i.e to get the sum of all the successful batches at the last as shown in the below sample output

Code:
 
Enter Date: 2013/03/07
 
Batches that were successful  2013/03/07 between 00:00:00 and 00:29:59: 0
Batches that were successful  2013/03/07 between 00:30:00 and 00:59:59: 0
Batches that were successful  2013/03/07 between 01:00:00 and 01:29:59: 0
Batches that were successful  2013/03/07 between 01:30:00 and 01:59:59: 0
Batches that were successful  2013/03/07 between 02:00:00 and 02:29:59: 0
Batches that were successful  2013/03/07 between 02:30:00 and 02:59:59: 0
Batches that were successful  2013/03/07 between 03:00:00 and 03:29:59: 0
Batches that were successful  2013/03/07 between 03:30:00 and 03:59:59: 0
Batches that were successful  2013/03/07 between 04:00:00 and 04:29:59: 0
Batches that were successful  2013/03/07 between 04:30:00 and 04:59:59: 0
Batches that were successful  2013/03/07 between 05:00:00 and 05:29:59: 0
Batches that were successful  2013/03/07 between 05:30:00 and 05:59:59: 0
Batches that were successful  2013/03/07 between 06:00:00 and 06:29:59: 0
Batches that were successful  2013/03/07 between 06:30:00 and 06:59:59: 0
Batches that were successful  2013/03/07 between 07:00:00 and 07:29:59: 0
Batches that were successful  2013/03/07 between 07:30:00 and 07:59:59: 0
Batches that were successful  2013/03/07 between 08:00:00 and 08:29:59: 0
Batches that were successful  2013/03/07 between 08:30:00 and 08:59:59: 0
Batches that were successful  2013/03/07 between 09:00:00 and 09:29:59: 0
Batches that were successful  2013/03/07 between 09:30:00 and 09:59:59: 0
Batches that were successful  2013/03/07 between 10:00:00 and 10:29:59: 0
Batches that were successful  2013/03/07 between 10:30:00 and 10:59:59: 0
Batches that were successful  2013/03/07 between 11:00:00 and 11:29:59: 0
Batches that were successful  2013/03/07 between 11:30:00 and 11:59:59: 0
Batches that were successful  2013/03/07 between 12:00:00 and 12:29:59: 0
Batches that were successful  2013/03/07 between 12:30:00 and 12:59:59: 0
Batches that were successful  2013/03/07 between 13:00:00 and 13:29:59: 0
Batches that were successful  2013/03/07 between 13:30:00 and 13:59:59: 0
Batches that were successful  2013/03/07 between 14:00:00 and 14:29:59: 0
Batches that were successful  2013/03/07 between 14:30:00 and 14:59:59: 0
Batches that were successful  2013/03/07 between 15:00:00 and 15:29:59: 0
Batches that were successful  2013/03/07 between 15:30:00 and 15:59:59: 0
Batches that were successful  2013/03/07 between 16:00:00 and 16:29:59: 0
Batches that were successful  2013/03/07 between 16:30:00 and 16:59:59: 0
Batches that were successful  2013/03/07 between 17:00:00 and 17:29:59: 0
Batches that were successful  2013/03/07 between 17:30:00 and 17:59:59: 0
Batches that were successful  2013/03/07 between 18:00:00 and 18:29:59: 0
Batches that were successful  2013/03/07 between 18:30:00 and 18:59:59: 0
Batches that were successful  2013/03/07 between 19:00:00 and 19:29:59: 0
Batches that were successful  2013/03/07 between 19:30:00 and 19:59:59: 0
Batches that were successful  2013/03/07 between 20:00:00 and 20:29:59: 0
Batches that were successful  2013/03/07 between 20:30:00 and 20:59:59: 0
Batches that were successful  2013/03/07 between 21:00:00 and 21:29:59: 0
Batches that were successful  2013/03/07 between 21:30:00 and 21:59:59: 37
Batches that were successful  2013/03/07 between 22:00:00 and 22:29:59: 55
Batches that were successful  2013/03/07 between 22:30:00 and 22:59:59: 118
Batches that were successful  2013/03/07 between 23:00:00 and 23:29:59: 65
Batches that were successful  2013/03/07 between 23:30:00 and 23:59:59: 49
----------------------------------------------------------------
Total Batches that were successful  on 2013/03/07: 324
----------------------------------------------------------------

Last edited by rpm120; 10-04-2013 at 01:52 PM..
# 10  
Old 10-04-2013
With the scripts I have given you and the explanation of what one of the scripts does, can you tell us where changes need to be made to the script to add the additional three lines of output at the end?

How close can you come to getting the script to produce the output you want?
This User Gave Thanks to Don Cragun For This Post:
rpm120
# 11  
Old 10-07-2013
Lightbulb
Quote:
Originally Posted by Don Cragun
With the scripts I have given you and the explanation of what one of the scripts does, can you tell us where changes need to be made to the script to add the additional three lines of output at the end?

How close can you come to getting the script to produce the output you want?
This is how I could take it forward

Code:
 
#!/bin/ksh
printf "Enter the date (YYYY/MM/DD): "
read dd
awk -F '[ :]' -v dd="$dd" '
BEGIN { fmt = "Batches that were successful on %s between %02d:%02d:00 " \ 
"and %02d:%02d:59 : %d\n"
}
$1 == dd && /successful/ { s[$2 + 0, $3 > 29]++ }
END {   for(h = 0; h < 24; h++)
for(m = 0; m < 2; m++)
printf(fmt, dd, h, m * 30, h, m * 30 + 29, s[h, m])
}' log >> report.txt
 
awk -F ":" -v dd="$dd" '{x += $NF} END {print "Total Batched That Were Successful on $dd :" x}' report.txt

# 12  
Old 10-07-2013
Please get into the habit of using indentation to show the structure of your code. It makes it a lot easier for humans to see what you're trying to do even if the awk interpreter (and the Korn shell) don't much care about spacing in the scripts they process (as long as reserved words are separated as required by their lexical analyzers).

I guess I confused you by separating the format string used by printf() to format its output from the printf() statement itself. Please look at your system's man page for the awk utility and look closely at its description of the awk printf() function.

As had been mentioned before, $dd in awk is not a reference to an awk variable named dd, it is a reference to the contents of the field specified by the numeric value of the contents of the awk variable dd. References to variables in the awk programming language and references to variables in the shell programming language are different. So, $dd (when dd contains 2013/03/07) is not valid; but, even if it was, $dd inside quotes in the format string will print the literal string $dd not the contents of the dd'th field in the current input line (which has no defined meaning in the END clause).

Try the following:
Code:
#!/bin/ksh
printf "Enter the date (YYYY/MM/DD): "
read dd
awk -F '[ :]' -v dd="$dd" '
$1 == dd && /successful/ {
        s[$2 + 0, $3 > 29]++
        t++
}
END {   for(h = 0; h < 24; h++)
                for(m = 0; m <= 1; m++)
                        printf("Batches that were successful on %s between %02d:%02d:00 and %02d:%02d:59 : %d\n",
                                dd, h, m * 30, h, m * 30 + 29, s[h, m])
        dl =  "----------------------------------------------------------------"
        printf("%s\nTotal Batches that were sucessful on %s: %d\n%s\n",
                dl, dd, t, dl)
}' log

I would normally split that format string into two parts (so my script would be easily readable on an 80 column output device:
Code:
                        printf("Batches that were successful on %s between " \
                                "%02d:%02d:00 and %02d:%02d:59 : %d\n",
                                dd, h, m * 30, h, m * 30 + 29, s[h, m])

but I didn't want to confuse you by splitting the format string into two parts.
This User Gave Thanks to Don Cragun For This Post:
rpm120
# 13  
Old 10-08-2013
Quote:
Originally Posted by Don Cragun
Please get into the habit of using indentation to show the structure of your code. It makes it a lot easier for humans to see what you're trying to do even if the awk interpreter (and the Korn shell) don't much care about spacing in the scripts they process (as long as reserved words are separated as required by their lexical analyzers).

I guess I confused you by separating the format string used by printf() to format its output from the printf() statement itself. Please look at your system's man page for the awk utility and look closely at its description of the awk printf() function.

As had been mentioned before, $dd in awk is not a reference to an awk variable named dd, it is a reference to the contents of the field specified by the numeric value of the contents of the awk variable dd. References to variables in the awk programming language and references to variables in the shell programming language are different. So, $dd (when dd contains 2013/03/07) is not valid; but, even if it was, $dd inside quotes in the format string will print the literal string $dd not the contents of the dd'th field in the current input line (which has no defined meaning in the END clause).

Try the following:
Code:
#!/bin/ksh
printf "Enter the date (YYYY/MM/DD): "
read dd
awk -F '[ :]' -v dd="$dd" '
$1 == dd && /successful/ {
        s[$2 + 0, $3 > 29]++
        t++
}
END {   for(h = 0; h < 24; h++)
                for(m = 0; m <= 1; m++)
                        printf("Batches that were successful on %s between %02d:%02d:00 and %02d:%02d:59 : %d\n",
                                dd, h, m * 30, h, m * 30 + 29, s[h, m])
        dl =  "----------------------------------------------------------------"
        printf("%s\nTotal Batches that were sucessful on %s: %d\n%s\n",
                dl, dd, t, dl)
}' log

I would normally split that format string into two parts (so my script would be easily readable on an 80 column output device:
Code:
                        printf("Batches that were successful on %s between " \
                                "%02d:%02d:00 and %02d:%02d:59 : %d\n",
                                dd, h, m * 30, h, m * 30 + 29, s[h, m])

but I didn't want to confuse you by splitting the format string into two parts.

Great Thanks for your time, comments and solution...!!! Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Script to search log file for last 15 mins data

Hi All, I have an issue which I'm trying to understand a way of doing, I have several nodes which contain syslog events which I want to force trigger an email initially (eventually leading to another method of alerting but to start with an email). Basically the syslog file will have hours worth... (6 Replies)
Discussion started by: mutley2202
6 Replies

2. UNIX for Dummies Questions & Answers

Extract only the data from ksh script running netezza query

Hi I searched this forum before posting the question, but couldnt find it, the issue i'm facing is, i'm trying to select a column from a netezza table from a korn shell script, but the query runs var=$(nzodbcsql -q "select MAX(millcount) from table1";) echo $var it returns the value like... (10 Replies)
Discussion started by: maximus_jack
10 Replies

3. Shell Programming and Scripting

Run Bash Script thrice & then interval for 10 mins.

Hi... I am very new to shell scripting. I have written a script with help of this forum and some googling and it works the way I want it to. Currently this script checks for my SIP trunk registration every 5 seconds, if registration is not available then it reboots my router through telnet... (4 Replies)
Discussion started by: jeetz
4 Replies

4. Shell Programming and Scripting

AWK counting interval / histogram data

My data looks like this: frame phi psi 0 68.466774 -58.170494 1 75.128593 -51.646816 2 76.083946 -64.300102 3 77.578056 -76.464218 4 63.180199 -76.067680 5 77.203979 -58.560757 6 66.574913 -60.000214 7 73.218269 -70.978203 8 70.956879 -76.096558 9 65.538872 -76.716568... (19 Replies)
Discussion started by: chrisjorg
19 Replies

5. Shell Programming and Scripting

Averaging data every 30 mins using AWK

A happy Monday to you all, I have a .csv file which contains data taken every 5 seconds. I want to average these 5 second data points into 30 minute averages! date co2 25/06/2011 08:04 8.31 25/06/2011 08:04 8.32 25/06/2011 08:04 8.33... (18 Replies)
Discussion started by: gd9629
18 Replies

6. Shell Programming and Scripting

KSH Script -- loop and data copy question

I am trying to write a script that will allow me to train others with commands that I run manually by only allowing the exact command before continuing onto the next set of commands. Here is where I come into an issue. I have changed the directories for this post. Software we run creates files... (2 Replies)
Discussion started by: hurtzdonut
2 Replies

7. Shell Programming and Scripting

How to Get 60 days Old date from current date in KSH script

Hi i am writing a cron job. so for it i need the 60 days old date form current date in variable. Like today date is 27 jan 2011 then output value will be stote in variable in formet Nov 27. i am using EST date, and tried lot of solution and see lot of post but it did not helpful for me. so... (3 Replies)
Discussion started by: Himanshu_soni
3 Replies

8. Shell Programming and Scripting

Need Help for interval date

Hello, I need help about a shell script I have a text file with this fields: 2009/01/19 09:33:35: --> ---ORA-28817: PL/SQL function returned an error. 2009/01/19 09:33:35: --> ---ORA-28817: PL/SQL function returned an error. 2009/01/19 09:33:35: --> ---ORA-28817: PL/SQL function returned an... (4 Replies)
Discussion started by: giofai
4 Replies

9. Shell Programming and Scripting

To extract data of a perticular interval (date-time wise)

I want a shell script which extract data from a log file which contains date and time-wise data and i need the data for a perticular interval of time...what can i do??? (3 Replies)
Discussion started by: abhishek27
3 Replies

10. Shell Programming and Scripting

Parsing and getting data from XML file using ksh script

Hi All, I have a xml file for example as described below <xml> <address> <street><street> <address/> <isbn>426728783932020308393930303</isbn> <book> <name> </name> </book> . . . </xml> My problem is to get the isbn number from the above described file using ksh script. Could... (6 Replies)
Discussion started by: vinna
6 Replies
Login or Register to Ask a Question