UNIX for Beginners Questions & Answers

Hi,
I have an unexpected reboot happening on a Debian 9.9 server.
Yesterday 2019-12-01 at 8:30:34 a reboot happened without me or my team being aware:

Code:

/var/log/syslog:Dec      1 08:30:34 xxxx shutdown[6027]: shutting down for system reboot
/var/log/syslog:Dec      1 08:30:34 xxxx init: Switching to runlevel: 6

I see nothing in any other log file.
What should I do to investigate the cause of the reboot?
Regards
Santiago

Does the output from last show you who (and where from) people were logged in at that time? Look especially for lines that say down in them.

Does that help you?


It might help in future if you logged all commands run by root to the syslog. Anything run by sudo may well be in /var/log/secure already.




Kind regards,
Robin

Also check cron logs to be sure nothing added to initiate the reboot.

Were there any updates prior to the reboot, as in updates to physical hardware?

As root run history to get root's command history.
Check /var/log/secure for system access (through PAM).
/var/log/su* might be special logs fo su/sudo.

Even if it does not directly help in this case, I'd like to mention Linux Audit, because it's really interesting:

One can audit all commands entered by anyone with these 2 rules:

Code:

auditctl -a always,exit -F arch=b32 -S execve -F key=audit_all_commands
auditctl -a always,exit -F arch=b64 -S execve -F key=audit_all_commands

To make it persistent, you need to place the commands in the auditd-rule files.(For Red Hat they are located in /etc/audit/rules.d)

...and then use ausearch for investigation of /var/log/audit/audit.log, or just grep it.

Interesting thing about Linux-Audit:

If you do su or sudo Linux-Audit tracks your original user anyway.

And for this investigation it won't help you: Because you need to have the rules in place before the event to be investigated happens.

Here's an example of the two most interesting lines of what you might have in the log(/var/log/audit/audit.log):

Code:

type=SYSCALL msg=audit(1575314015.062:20): arch=c000003e syscall=59 success=yes exit=0 a0=55c7fbbc4e20 a1=55c7fbbc43a0 a2=55c7fbbc86a0 a3=2 items=2 ppid=1 pid=832 auid=1004 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="reboot" exe="reboot" key="audit_all_commands"
type=EXECVE msg=audit(1575314015.062:20): argc=1 a0="reboot"

Auditd is very noisy in logging. That means one possibly has to dig a lot log records.

As you see above the key audit_all_commands is in the line. So you can grep for your configured string. And uid in the first line shows 0 which is root. But the auid-value shows 1004, which is the real user id(which originally logged on to the system before using su or sudo. So it's a good idea to restrict direct root access and only allow sudo for becoming root).

The second line is linked with the first line through the audit-id field(audit(1575314015.062:20)) and shows you the linked command with parameters executed.

What can you do with Linux-Audit?


I worked the last 3 month with it and did a lot with it. For example i created a little perl script which uses inotify & git & audit to log diffs of config file changes into audit log which could then be tracked down to the causing user accounts.

Here's the perl script, if someone cares: [Perl] #!/usr/bin/perl # # - Track files in /etc # - Create audit log - Pastebin.com