UNIX for Beginners Questions & Answers

Hi,

I am totally a newbie to any programming languages and I just started an entry level job in an IT company. One of my recent tasks is to create a script that is able to show the log file of linux service (i.e. ntpd service)
lets say, if I run my script ./test.sh, the output should be similar to :

Code:

2019-02-26 09:15:00 +1100,Success,ntpd,restart,Service Successfully Restart

I dont know if I didnt get the right requirement or something, I found a lot of scripts online but none of them worked for my case. One thing that I don't understand is, do I firstly need to have an existing .log file that has all the ntpd service history? I check var/log/ntpstats but there is no log files.

I would really appreciate if anyone could give me some ideas, if you need any clarifications please let me know.

Thanks very much

xiao

Let us take a step back: programming is about automating tasks and the problem is NOT to write the program but to define as succinctly as possible WHAT it is you want to automate. Running faster will get you nowhere if the direction you run to is wrong.

So, let us think:

1) what do you want your log to look like? You already showed a sample line and, agreed, this might be a good start. But where will the entries come from and how will they be organised? By that i mean:

Code:

2019-02-26 09:15:00 +1100,Success,ntpd,restart,Service Successfully Restart

The first part, the date and time is clear. But "Success,ntpd,restart" and "Service Successfully Restart" is essentially the same information, no? Suppose you have this (and maybe many other) scripts running and they all produce 5000 entries a day. You don't want to read them all but may want a script to "read" them, filter out all the "normal" stuff and only tell you about problems. For such a script to be easily written you want to log entries in a homgenous format which lends itself well to be read by a program. Like this:

Code:

$(date and time in fixed format): $(success status), $(service name), $(activity)

We already had date and time, "$(success status)" could be one of "success", "error", "warning", then "$(service name) could be "ntpd" for your script or something else for a similar script and finally "$(activity)" would be whatever it is your script is logging at that time. A possible log would then look like:

Code:

2019-02-26 09:15:00 +1100,Success,ntpd,restart
2019-02-26 09:16:00 +1100,Error,ntpd,other-activity
2019-02-26 09:17:00 +1100,Warning,ntpd,something else
2019-02-26 09:18:00 +1100,Success,ntpd,restart
...

2) The "fields" i did describe here are just for example. You need to define yourself what you need to log for each process. The better your definition is the easier it will be to apply that to all sorts of services to be logged and always end up with the same format. Ideas for which information might (or might not) be interesting to log is:

Code:

$(date and time in fixed format)
$(success status)
$(service name)
$(activity)
$(exact command issued)
$(return code/output of that command)

But you can easily extend this list and might identify - depending on your environment - many other possibly interesting information.

3) single log versus multiple logs
That brings us to the next question: if you log many services there are two possible strategies. You write one log for every service (i.e. ntpd.log, ftpd.log, sshd.log, etc. or you write one log for all. If you write separate logs for each service you do not need to log the service name in every entry because i.e. what else than "ntpd" as a service name is supposed to stand in the file ntpd.log? On the other hand it might be a good idea to have only one log file where all services are logged. You only have one place to look at and if one service fails and affects other services this is easier to see in correlation because the entries are already sorted by time. What is "better", single or separate logs, depends on your environment and your goals. There is no generally right or wrong answer. Again, analyse carefully your environment and apply good reasoning.

4) separate error log
A final point: it sometimes makes sense to write a separate log for log entries of the "error" condition. That is: "success"-type entries go to the normal log, "warning"-type entries also go to the normal log and "error"-type entries go to the normal AND to a separate error-log. You might then have (if you have separate logs for each service) a ntpd.log and a ntpd.errlog, a ftpd.log and a ftpd.errlog, etc., or if you have a single log you have a system.log and a system.errlog. The log writing becomes a bit more complicated but if errors are usually rare (again, what is considered an "error" depends on your environment, so you will have to analyse carefully) you can just check the size of the error log and if it is empty you know at a glance that no error has happened. You may need to investigate more closely only if the error-log is not empty.

I suggest you try to come up with a plan on what you want to do once you thought it through thoroghly and then we discuss how to implement your plan.

I hope this helps.

bakunin

Hey bakunin,


Many thanks for your reply!


the following is my plan and let me know if anything doesnt make sense :


1. Ideally the script should print out the log file in the following format:
2019-02-26 09:15:00 +1100,Success,ntpd,restart 2019-02-26 09:16:00 +1100,Error,ntpd,other-activity

But if I am really stuck,as long as the "$(success status),$(service name),$(activity)" are in the output then it will still be fine, someone will extend the script later on..
2. and the fields that I want in the log should include, the last three fields are the minimum and the date is optional depending on the complexity:

$(date and time in fixed format), $(success status), $(service name), $(activity)



3. single log vs multiple logs: I shouldve clarified earlier that the only service I need to work on is ntpd. So a single log would be fine for me.


4. I am not sure if it will be more complicated to write a separate error log since I only need to focus on one service which is ntpd.



Let me know if it makes sense to move on to the next step. thanks again for your help!

First off: i apologise for dragging this out a bit. Once you are experienced you could probably do all the reasoning we are going through here in a few seconds of thought. But since you lack this experience and i want to show you how you tackle this sort of problems i show you and discuss every point applicable to any monitoring script. Some of what i will tell you will not be applicable to NTP because it is a very simply service. Others are more complicated and you will need this kind of reasoning, though.

So, bear with me - you might have come asking for a meal, i show you how to cook yourself.

All makes sense to me, save for one point:

Compare these two quotes. If there is only one service you want to monitor then why repeat the name of that service in every line? Do you really need that? Because the file will already be named "ntpd.log" or something such, so one would know that the messsages in there are about NTP or ntpd respectively, no?

Now, having identified the fields (that is: the type of information you want to log) you need to identify how this information will be comprised in detail. What do i mean by that?

Well, take "success status", for instance: Th simplest model would be to have to possible values: "success" or "failure" (or "error" or whatever you want to name it). You could also have three possible outcomes, "success", "warning" and "failure". You could even have more. Have a look at the documentation of syslog to get ideas. Basically syslog has "severity levels" which map to what i called "success status" here, seven of them. You specify a "threshhold severity" and everything "above" that severity will be logged, everything else will not.

Really? If you leave out the time stamp you have a message and don't know when that was. It is easy to do that but i wonder if it makes sense to do it this way.

OK, go over your list a seond time, plan on how many success status you want to have (and how they are called) and how you want to define them: What constitutes "success", what constitutes a "failure"? And how much has to go "quite not right" when you label the outcome a "warning". Once you do this we finally get around to writing the script.

I hope this helps.


bakunin