Red Hat

Have a question about doing some performance monitoring - how to approach it.

This is on RHEL 7.4

We are moving a rather large application to new hardware in the future. I would like to find a way to compared performance on the new hardware at different stages and at different times.

I'm familiar with the basic tools, such as top - and I've found an app called 'stress' that can be used to put various loads on the system.

But I am wanting to make a comparison on how the performance varies or does not vary after things like: Firmware upgrades, App installs, etc.

Any suggestions on a method/process or tool maybe that can assist with this? I can cap out the loads with 'stress' but I'm not sure what that will tell me. My first thought was to run the app with a certain set of parameters and then compare future runs using the same set of parameters. I would need or like it to be able to log results at set intervals via cron or something similar.

But looking for suggestions

Thanks!

Hi,


I suggest you to get some monitoring tool which collects alle the performance data you need. There are several out there, commercial and open source alike:

I'm a check_mk(Infrastructure & Application Monitoring | Check_MK) user(i use the free version for about 8000 single services) and I like it very much, because there's some magic already built-in: Discover several basic performance values automatically and if additional tools are installed, it automatically uses them. (Temperature sensors, cpu load, cpu usage, memory usage, hard disk i/o, network i/o, network errors, hard disk smart data(needs to be enabled)).

In default mode it creates graphs which are configured for highest granularity for the last minutes and shows overall performance development over long time ranges.

Here is a standard graph for filesystem usage trend, growth/shrink and inodes usage:

https://webmail.megabert.de//data/ha...row_inodes.png

The webinterface has a pleasant user experience.


Look here for a live demo:

Demoserver | Check_MK

Check out Prometheus - Monitoring system & time series database
A free and open source product.
Nice integration, drawing all kinds of stuff, inside jvm via jmx for instance etc.

Alerting is also supported as a separate process.
For dashboards and stuff grafana is to be used with it.

After exploring a lot of option(s) via virtualization, this one struck my soft spot, so it will hopefully be used

Regards
Peasant.

As it seems, Prometheus ist the current hot shit in monitoring(no offence). Some former colleagues of mine are very fond of that tech too.

My idea of Prometheus is, that it is real cool pile of nuts and bolts, where you can build most advanced planes of. Since know almost nothing about Prometheus, this is very likely a wrong view of mine.

Peaseant, if you like, I'd like to hear, what it is, that fascinates you about Prometheus.

I like check_mk because - staying at my metaphor - it IS a quite nice plane, which is ready to take off after single command installation.


Update

A short investigation of mine seems to confirm my current view. P. is a sophisticated and powerful piece of software which requires considerable learning effort to get it into work and then you may scalably process a whole lot of servers. But if you only have a fistful of servers that's a far too big gun for the task.

Well, i cannot say fascinated, since i've used it on home computer (fairly complex smartos environment with dozens of machines).

Currently i use Cacti, since i do not wish not to use agent software if
possible.
I just don't trust anyone
Check out the new Cacti, it's quite nice nowdays and even looks fancy.
SNMP is quite enough for 95% of monitoring. To bad is not widely accepted.

For else i have (for now) crontab with various stats* and plot.
This is for drill down analysis if required.

As for Prometheus, i actually love those kind of products you mention.
A lot of options and features, complex.

It's a kind of product you think upfront what you want, then fire and forget.

As for features - if we talk whole stack....

I like API - get graph with URL, inputting what you want directly into url.
When you draw a graph from gui it gives you a nice URL to use as well (or send around).

Language is not strange to me (GO), i personally like it and explored it a bit, should work on just any unix out there.

Alerting API is quite broad. From mails to endpoint API(s) to various stuff.

Last but not least, official client libs for java, ruby, go and scala.
Other unofficial in large numbers.
Include metrics in your apps is something i have yet to see from others
This can lower the debug time and/or expensive software such as dynatrace.

Since you are a ruby fan, take it for a spin

It's really fast with nginx infront(reverse proxy and internal load balancer), even on couple of AMD FM1 cores @ home, running that and dozens of smartos vms in virtualbox.

I'm willing to suggest this product to replace Cacti completely in my work environment, gradually for new projects.

I'm writing this as totally not affiliated with Prometheus or Cacti in any way.

Regards
Peasant.