I've gotta post soon what I'm doing.

I also am googling some things

---------- Post updated at 02:51 AM ---------- Previous update was at 12:25 AM ----------

hey guys sorry for the mess I made in my previous post.
Here goes:

program will do the following

#1 get variables from a db and assign to a shell variables.
#2 post those variables to logs
#3 every hour the script runs it saves logs
#4 these logs will go to a log file for each day running during business hours
#5 Now the tricky part
I have to for each business day the do calculations like percentage calculations on variables collected in logs for each hour during the day at the end of the day. and eventually a summary of all data points collected during the month until the month.

The datapoints I collect to variables are only pure calculations from system performance, like response times, so only numbers.

Heres what I am thinking.

Collected logs will look like:

Code:

############################
2014-08-02 08:00:00 -VAR1: 1151 VAR2: 15151 ...
2014-08-02 09:00:00 -VAR1: 1151 VAR2: 15151 ...
2014-08-02 10:00:00 -VAR1: 1151 VAR2: 15151 ...
2014-08-02 11:00:00 -VAR1: 1151 VAR2: 15151 ...
2014-08-02 12:00:00 -VAR1: 1151 VAR2: 15151 ...

I will have many variables per hour max like 5 varibles per hour.

-these variable will be needed to collected for calculations over time.
1: aggregation mathematics for speed of the variables.
2: these variables will be collected to do aggregation, I would prefer to save them to logs like after the Vars I can do a overall calculation...

Code:

2014-08-02 08:00:00 -VAR1: 1151 VAR2: 15151 VAR3: 1151 VARTOTAL: 11515151

vartotal will be my tool to use for the percentage calculations or any necessary calculations I need going forward.

More info:
I am collecting system performance info and response times, aggregates will be calculated to see how our system performs over time...

The datapoints are 100% needed to be aggregated for our needs.

-how do you guys suggest I grab variables saved to logs by date? and how do I point logs to search datapoints from the logs by saying to search date hour.

#1 possibilty I think that is better for me.
1st script writes the logs file and does the system speed gatherring for those wonderring still its a combination of DB + linux reponse times saved to variables and >> to log and saved by the hour during business hours.

Anyways the logs will be like I explained one will be for my maintainence chekcs and other to save datapoitns like this

#2 The logs collected will have TOTAL column and other calculation columns...

Code:

2014-08-02 08:00:00 -VAR1: 1151 VAR2: 15151 VAR3: 1151 VARTOTAL: 11515151

so that I can use the points over time and do calculations.

I am searching for things I need to get this done anyone want to add in

#2 the second script: grabs total saved calculations for the logs for the month and then produces my aggregations.
one thing confusing me is how to I add to the script a way to search and collect data by date, as over time I may need to modify it to lets save give me a summary of months 1 through 3.

I will need too separate the functionality as the more I think I need too do calculations like over custom times as business needs require.

Any help I can get mates would be appreciated.

Anyone need more information from me like what I have done so far please wait a few hours as I am deploying something I got help making from here yesterday

---------- Post updated at 07:44 AM ---------- Previous update was at 02:51 AM ----------

hey all I will post a working script I got so far!

I need to make the second script that will do aggregation from grabbing all datapoints from the logs up till a month or what I define in the logs.

NOONE worry about the script that does the calculations and gathers variables its done...
#2 how do I aggregate over time data collected through a script that reads of a log table from lets say data collected over a weekeday or month

---------- Post updated at 11:07 AM ---------- Previous update was at 07:44 AM ----------

If I do my calculations on many variables in my script and the logs produces lets say these logs:

Code:

####LOG FILE ###
2014-08-04
$Date=2014-08-04  $hour
$Date2=2014-08-04  $onehourfromnow

variables collected (...)
sumations (....)
totalcolumns(...)

#1 My main script produces this log NOW I need my second script help!

This log file will be written as >> thus all results saved and over time I should have many hrs saved across many days.



Script #2, how do I take these variables saved by each hour and day I schedule the program and THEN do massive calculations and summations

For instance I will have 100 seperate of the LOG file code above saved by month end and then I want to be able to tell my script start searching from month and grab every variable you see and create as many temporary variables as needed to do the necessary calculations.

anyone following what I am saying?

i am thinking as mart grep solution must exist, grep each line start this date here and start picking all hours until get to other line etc...

this is where I am stuck... how do I summarize my results by the date variable I chose, month or week etc...

remember its a seperate program then script 1 which produces the log files!