Shell Programming and Scripting

Hey guys, I am working on a script that needs to grab variables from a log file. The script will run morning 9 to 5pm and save variables for each run every hour, these results I will be aggregating at the end of the day.

I am thinking I will be placing the date on each entry in the log every hour, so I think I can grep each date and then go from there.

Can anyone help me with the following?

I am in the train right now so when I get home I will add more ideas.

P.s. If anyone is confused I need to conduct calculations on the enteries on each of the log dates. I will


-----------------------------------------------------------------------------------------

program will do the following

#1 get variables from a db and assign to a shell variables.
#2 post those variables to logs
#3 every hour the script runs it saves logs
#4 these logs will go to a log file for each day running during business hours
#5 Now the tricky part
I have to for each business day the do calculations like percentage calculations on variables collected in logs for each hour during the day at the end of the day. and eventually a summary of all data points collected during the month until the month.

The datapoints I collect to variables are only pure calculations from system performance, like response times, so only numbers.

Heres what I am thinking.

Collected logs will look like:

Code:

:
 ############################ 2014-08-02 08:00:00 -VAR1: 1151 VAR2: 15151 ... 2014-08-02 09:00:00 -VAR1: 1151 VAR2: 15151 ... 2014-08-02 10:00:00 -VAR1: 1151 VAR2: 15151 ... 2014-08-02 11:00:00 -VAR1: 1151 VAR2: 15151 ... 2014-08-02 12:00:00 -VAR1: 1151 VAR2: 15151 ...

I will have many variables per hour max like 5 varibles per hour.

-these variable will be needed to collected for calculations over time.
1: aggregation mathematics for speed of the variables.
2: these variables will be collected to do aggregation, I would prefer to save them to logs like after the Vars I can do a overall calculation...

Code:

 2014-08-02 08:00:00 -VAR1: 1151 VAR2: 15151 VAR3: 1151 VARTOTAL: 11515151

vartotal will be my tool to use for the percentage calculations or any necessary calculations I need going forward.

More info:
I am collecting system performance info and response times, aggregates will be calculated to see how our system performs over time...

The datapoints are 100% needed to be aggregated for our needs.

-how do you guys suggest I grab variables saved to logs by date? and how do I point logs to search datapoints from the logs by saying to search date hour.

#1 possibilty I think that is better for me.
1st script writes the logs file and does the system speed gatherring for those wonderring still its a combination of DB + linux reponse times saved to variables and >> to log and saved by the hour during business hours.

Anyways the logs will be like I explained one will be for my maintainence chekcs and other to save datapoitns like this

#2 The logs collected will have TOTAL column and other calculation columns...

Code:

 2014-08-02 08:00:00 -VAR1: 1151 VAR2: 15151 VAR3: 1151 VARTOTAL: 11515151

so that I can use the points over time and do calculations.

I am searching for things I need to get this done anyone want to add in

#2 the second script: grabs total saved calculations for the logs for the month and then produces my aggregations.
one thing confusing me is how to I add to the script a way to search and collect data by date, as over time I may need to modify it to lets save give me a summary of months 1 through 3.

I will need too separate the functionality as the more I think I need too do calculations like over custom times as business needs require.

Any help I can get mates would be appreciated.

Anyone need more information from me like what I have done so far please wait a few hours as I am deploying something I got help making from here yesterday

to help explain what I am trying to accomplish! was on train getting home! Thanks for the patience guys

If you can't show us some sample input and show us what you are hoping to get out of it, there is very little we can do to help you.

Show us your input. Show us the output you want. Show us the code you have tried and explain what isn't working for you.

Don't you understand he is unable to think a way out. If he could write the code why would he come to ask for help. He has an idea but not the necessary coding skills. So all is he asking for some code snippets. If he himself could write the code and counters some error he would Google it not post it here. I have faced this kind of situation once and it's really very frustrating.

I've gotta post soon what I'm doing.

I also am googling some things

---------- Post updated at 02:51 AM ---------- Previous update was at 12:25 AM ----------

hey guys sorry for the mess I made in my previous post.
Here goes:

program will do the following

#1 get variables from a db and assign to a shell variables.
#2 post those variables to logs
#3 every hour the script runs it saves logs
#4 these logs will go to a log file for each day running during business hours
#5 Now the tricky part
I have to for each business day the do calculations like percentage calculations on variables collected in logs for each hour during the day at the end of the day. and eventually a summary of all data points collected during the month until the month.

The datapoints I collect to variables are only pure calculations from system performance, like response times, so only numbers.

Heres what I am thinking.

Collected logs will look like:

Code:

############################
2014-08-02 08:00:00 -VAR1: 1151 VAR2: 15151 ...
2014-08-02 09:00:00 -VAR1: 1151 VAR2: 15151 ...
2014-08-02 10:00:00 -VAR1: 1151 VAR2: 15151 ...
2014-08-02 11:00:00 -VAR1: 1151 VAR2: 15151 ...
2014-08-02 12:00:00 -VAR1: 1151 VAR2: 15151 ...

I will have many variables per hour max like 5 varibles per hour.

-these variable will be needed to collected for calculations over time.
1: aggregation mathematics for speed of the variables.
2: these variables will be collected to do aggregation, I would prefer to save them to logs like after the Vars I can do a overall calculation...

Code:

2014-08-02 08:00:00 -VAR1: 1151 VAR2: 15151 VAR3: 1151 VARTOTAL: 11515151

vartotal will be my tool to use for the percentage calculations or any necessary calculations I need going forward.

More info:
I am collecting system performance info and response times, aggregates will be calculated to see how our system performs over time...

The datapoints are 100% needed to be aggregated for our needs.

-how do you guys suggest I grab variables saved to logs by date? and how do I point logs to search datapoints from the logs by saying to search date hour.

#1 possibilty I think that is better for me.
1st script writes the logs file and does the system speed gatherring for those wonderring still its a combination of DB + linux reponse times saved to variables and >> to log and saved by the hour during business hours.

Anyways the logs will be like I explained one will be for my maintainence chekcs and other to save datapoitns like this

#2 The logs collected will have TOTAL column and other calculation columns...

Code:

2014-08-02 08:00:00 -VAR1: 1151 VAR2: 15151 VAR3: 1151 VARTOTAL: 11515151

so that I can use the points over time and do calculations.

I am searching for things I need to get this done anyone want to add in

#2 the second script: grabs total saved calculations for the logs for the month and then produces my aggregations.
one thing confusing me is how to I add to the script a way to search and collect data by date, as over time I may need to modify it to lets save give me a summary of months 1 through 3.

I will need too separate the functionality as the more I think I need too do calculations like over custom times as business needs require.

Any help I can get mates would be appreciated.

Anyone need more information from me like what I have done so far please wait a few hours as I am deploying something I got help making from here yesterday

---------- Post updated at 07:44 AM ---------- Previous update was at 02:51 AM ----------

hey all I will post a working script I got so far!

I need to make the second script that will do aggregation from grabbing all datapoints from the logs up till a month or what I define in the logs.

NOONE worry about the script that does the calculations and gathers variables its done...
#2 how do I aggregate over time data collected through a script that reads of a log table from lets say data collected over a weekeday or month

---------- Post updated at 11:07 AM ---------- Previous update was at 07:44 AM ----------

If I do my calculations on many variables in my script and the logs produces lets say these logs:

Code:

####LOG FILE ###
2014-08-04
$Date=2014-08-04  $hour
$Date2=2014-08-04  $onehourfromnow

variables collected (...)
sumations (....)
totalcolumns(...)

#1 My main script produces this log NOW I need my second script help!

This log file will be written as >> thus all results saved and over time I should have many hrs saved across many days.



Script #2, how do I take these variables saved by each hour and day I schedule the program and THEN do massive calculations and summations

For instance I will have 100 seperate of the LOG file code above saved by month end and then I want to be able to tell my script start searching from month and grab every variable you see and create as many temporary variables as needed to do the necessary calculations.

anyone following what I am saying?

i am thinking as mart grep solution must exist, grep each line start this date here and start picking all hours until get to other line etc...

this is where I am stuck... how do I summarize my results by the date variable I chose, month or week etc...

remember its a seperate program then script 1 which produces the log files!

Please show us a complete sample log file:

Code:

####LOG FILE ###
2014-08-04
$Date=2014-08-04  $hour
$Date2=2014-08-04  $onehourfromnow

variables collected (...)
sumations (....)
totalcolumns(...)

with the ...'s expanded and with more than one hour's data, and show us the report(s) you want to produce from that sample log file.

Hey Don,

The variable data can be anything really, I can save it any format necessary. and to any number of extra logs I need to be able to achieve my goal.

I can save it any format that will make it better for the over time aggregation aswell

like an exta log will save data in any way as the process goes on...
like so:

Code:

date.
hourstart 
#####
#####
#####
#####
#####
date.
hourstart2
#####
#####
#####
#####
#####
 date.
hourstart3
#####
#####
#####
#####
#####
(...)
date2
hourstart1
###
(...)

####-signs being the values colleted and it can go on to infinity really for each data set which right now I save to seperate logs, different checks go in different logs, for instance system speed one log, system space other logs and so on...

Just so you can understand better. I can save in any format and any extra number of logs that will make the aggregation and caclulstions I mentioned earlier easier.

So for instance right now I have it setup to just generate a log for the hourly checks. I am currently searching how I will approach aggregating and other calculations

The data I am saving is
1: a data set of system repsonse times
2: system related calculations
3: other data sets collected in relation to the DB software performance

all datasets will be seperate logs
the main dataset I will be collecting will be used for the aggregtion on number sets collected like so generally:

Code:

ForDateIsToday
ForEachHour
SystemResponset_to_testload1
SystemResponset_to_testload2
SystemResponset_to_testload3
SystemResponset_to_testload4
SystemResponset_to_testload5

each test load gets a result and then I save response time as a sort of system check.

Aggregating this data and doing my calculations on lets say a one month or over many months is important for me

-dates and related variables will be saved.

Don-if you can help me try to think of how I might aggregate over time and keeping in mind I can save the logs in any necessary format.

the main code I have is done and the logs I have now are ok in terms of SLAs
the aggregation and extras I mentioned are a plus point we need going forward upper management want logs that we can use for "calculations" that can tell is performance over time like avgs,speed chg rates etc.

if your confused about the logs still, just think of it as it can be anything like:

Code:

date,hourstart
-dataset relating to the logs below for the hour
#,#,#,#

I guess I'm too dense to understand how to help you. Every time you show us sample log file contents, the details change completely. With the latest details you've given us, I could suggest that you could use some code like:

Code:

awk '
/[.]$/ { ... # note start of new date }
/.../ { ... # do something }
/###/ { ... # do something else }
/???/ { ... # do who knows what }
END { ... # process accumulated data to produce report }
' log_file...

but I'm guessing that wouldn't be a big help to you. If you would like to show us some actual sample input data and an actual sample of the output you'd like to produce, we might be able to provide a better example of code that might help you reach your goal.