Login or Register to Ask a Question and Join Our Community


UNIX for Advanced & Expert Users

Error starting httpd via systemctl

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Error starting httpd via systemctl
# 1  
Old 12-16-2019
Error starting httpd via systemctl
Hi,


Most of our systems have the apache customized, and started manually via the command line. One eg :


Code:
[root@dev1-myeg system]# ps -aux | grep http
root      6278  0.0  0.0 112708   984 pts/2    S+   16:32   0:00 grep --color=auto http
root      6398  0.0  0.3 570636 14376 ?        Ss   Sep17   8:34 /usr/local/apache-2.4.33/bin/httpd -k start
apache   17671  0.0  5.5 2598496 209928 ?      Sl   Dec10   0:21 /usr/local/apache-2.4.33/bin/httpd -k start
apache   17672  0.0  5.5 2601972 209792 ?      Sl   Dec10   0:20 /usr/local/apache-2.4.33/bin/httpd -k start
apache   17673  0.0  5.3 2601972 201648 ?      Sl   Dec10   0:23 /usr/local/apache-2.4.33/bin/httpd -k start
apache   17799  0.0  5.5 2602232 209604 ?      Sl   Dec10   0:24 /usr/local/apache-2.4.33/bin/httpd -k start

To start the httpd service ,we use :


Code:
/usr/local/apache-2.4.33/bin/httpd -k start

However, we want to automate the service via systemd. Thus I modified the original script which is this :


Code:
[root@dev1-myeg system]# cat httpd24-httpd.service.bak
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=notify
EnvironmentFile=/opt/rh/httpd24/root/etc/sysconfig/httpd
ExecStart=/opt/rh/httpd24/root/usr/sbin/httpd-scl-wrapper $OPTIONS -DFOREGROUND
ExecReload=/opt/rh/httpd24/root/usr/sbin/httpd-scl-wrapper $OPTIONS -k graceful
ExecStop=/opt/rh/httpd24/root/usr/sbin/httpd-scl-wrapper $OPTIONS -k graceful-stop
# We want systemd to give httpd some time to finish gracefully, but still want
# it to kill httpd after TimeoutStopSec if something went wrong during the
# graceful stop. Normally, Systemd sends SIGTERM signal right after the
# ExecStop, which would kill httpd. We are sending useless SIGCONT here to give
# httpd time to finish.
KillSignal=SIGCONT
PrivateTmp=true

[Install]
WantedBy=multi-user.target
[root@dev1-myeg system]#

To this :


Code:
[root@dev1-myeg system]# cat httpd.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=notify
EnvironmentFile=/usr/local/apache-2.4.33/bin/httpd
ExecStart=/usr/local/apache-2.4.33/bin/httpd $OPTIONS -DFOREGROUND
ExecReload=/usr/local/apache-2.4.33/bin/httpd $OPTIONS -k graceful
ExecStop=/bin/kill -WINCH ${MAINPID}

# We want systemd to give httpd some time to finish gracefully, but still want
# it to kill httpd after TimeoutStopSec if something went wrong during the
# graceful stop. Normally, Systemd sends SIGTERM signal right after the
# ExecStop, which would kill httpd. We are sending useless SIGCONT here to give
# httpd time to finish.
KillSignal=SIGCONT
PrivateTmp=true

[Install]
WantedBy=multi-user.target
[root@dev1-myeg system]#

I then killed the currently running httpd process, to test if the script works. I get this error when I start the script :


HTTP process killed :


Code:
[root@dev1-myeg system]# ps -aux | grep http
root      7272  0.0  0.0 112708   984 pts/2    S+   16:44   0:00 grep --color=auto http

Starting the httpd service after it had been killed :



Code:
[root@dev1-myeg system]# systemctl start httpd.service
Warning: httpd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for httpd.service failed because a configured resource limit was exceeded. See "systemctl status httpd.service" and "journalctl -xe" for details.

I then edit the script to this (edited the command to start httpd) :


Code:
[root@dev1-myeg system]# cat httpd.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=notify
EnvironmentFile=/usr/local/apache-2.4.33/bin/httpd
#ExecStart=/usr/local/apache-2.4.33/bin/httpd $OPTIONS -DFOREGROUND
ExecStart=/usr/local/apache-2.4.33/bin/httpd -k start
ExecReload=/usr/local/apache-2.4.33/bin/httpd $OPTIONS -k graceful
ExecStop=/bin/kill -WINCH ${MAINPID}

# We want systemd to give httpd some time to finish gracefully, but still want
# it to kill httpd after TimeoutStopSec if something went wrong during the
# graceful stop. Normally, Systemd sends SIGTERM signal right after the
# ExecStop, which would kill httpd. We are sending useless SIGCONT here to give
# httpd time to finish.
KillSignal=SIGCONT
PrivateTmp=true

[Install]
WantedBy=multi-user.target
[root@dev1-myeg system]#

Trying to stop the service after the error :


Code:
[root@dev1-myeg system]# systemctl stop httpd.service
Warning: httpd.service changed on disk. Run 'systemctl daemon-reload' to reload units.

Still get error when trying to start the service :


Code:
[root@dev1-myeg system]# systemctl start httpd.service
Warning: httpd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job  for httpd.service failed because a configured resource limit was  exceeded. See "systemctl status httpd.service" and "journalctl -xe" for  details.

Trying to reload httpd :


Code:
[root@dev1-myeg system]# systemctl daemon-reload httpd.service
Invalid number of arguments.

Checking the output from httpd status :


Code:
[root@dev1-myeg system]# systemctl status httpd.service
- httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: failed (Result: resources)

Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: Failed to load environment files: Argument list too long
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service failed to run 'start' task: Argument list too long
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: Failed to start The Apache HTTP Server.
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: Unit httpd.service entered failed state.
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service failed.
Dec 16 16:44:35 dev1-myeg.int.myeg.com.my systemd[1]: Failed to load environment files: Argument list too long
Dec 16 16:44:35 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service failed to run 'start' task: Argument list too long
Dec 16 16:44:35 dev1-myeg.int.myeg.com.my systemd[1]: Failed to start The Apache HTTP Server.
Dec 16 16:44:35 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service failed.
Warning: httpd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
[root@dev1-myeg system]#

This is the output from journalctl -xe :


Code:
-- Logs begin at Tue 2019-12-10 05:01:01 +08, end at Mon 2019-12-16 17:40:02 +08. --
Dec 16 16:42:01 dev1-myeg.int.myeg.com.my systemd[1]: Started Session 133274 of user root.
-- Subject: Unit session-133274.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-133274.scope has finished starting up.
--
-- The start-up result is done.
Dec 16 16:42:01 dev1-myeg.int.myeg.com.my CROND[7051]: (root) CMD (cd /usr/local/apache31-MYEG/htdocs/myeg_new_ui && php artisan get:survey > /usr/local/apache31-MYEG/logs/getsurvey-cron.lo
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/pickup[6538]: 34C8D479D7F: uid=0 from=<root>
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/cleanup[4839]: 34C8D479D7F: message-id=<20191216084202.34C8D479D7F@dev1-myeg.int.myeg.com.my>
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 34C8D479D7F: from=<root@dev1-myeg.int.myeg.com.my>, size=2539, nrcpt=1 (queue active)
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/local[4841]: 34C8D479D7F: to=<root@dev1-myeg.int.myeg.com.my>, orig_to=<root>, relay=local, delay=0.41, delays=0.39/0/0/0.01, dsn=5.2.2, st
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/cleanup[4839]: 3B9CC47C211: message-id=<20191216084202.3B9CC47C211@dev1-myeg.int.myeg.com.my>
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 3B9CC47C211: from=<>, size=4538, nrcpt=1 (queue active)
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/bounce[6960]: 34C8D479D7F: sender non-delivery notification: 3B9CC47C211
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 34C8D479D7F: removed
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/local[4841]: 3B9CC47C211: to=<root@dev1-myeg.int.myeg.com.my>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (ca
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 3B9CC47C211: removed
Dec 16 16:42:15 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-16T16:42:15+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my polkitd[3254]: Registered Authentication Agent for unix-process:7087:779850084 (system bus name :1.269220 [/usr/bin/pkttyagent --notify-fd 5 --fall
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: Failed to load environment files: Argument list too long
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service failed to run 'start' task: Argument list too long
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: Failed to start The Apache HTTP Server.
-- Subject: Unit httpd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has failed.
--
-- The result is failed.
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: Unit httpd.service entered failed state.
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service failed.
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my polkitd[3254]: Unregistered Authentication Agent for unix-process:7087:779850084 (system bus name :1.269220, object path /org/freedesktop/PolicyKit
Dec 16 16:42:23 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-16T16:42:23+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http:/
Dec 16 16:42:55 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-16T16:42:55+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server
Dec 16 16:43:01 dev1-myeg.int.myeg.com.my systemd[1]: Started Session 133275 of user root.
-- Subject: Unit session-133275.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-133275.scope has finished starting up.
--
-- The start-up result is done.
Dec 16 16:43:01 dev1-myeg.int.myeg.com.my CROND[7144]: (root) CMD (cd /usr/local/apache31-MYEG/htdocs/myeg_new_ui && php artisan get:survey > /usr/local/apache31-MYEG/logs/getsurvey-cron.lo
Dec 16 16:43:02 dev1-myeg.int.myeg.com.my postfix/pickup[6538]: 36908479D7F: uid=0 from=<root>
Dec 16 16:43:02 dev1-myeg.int.myeg.com.my postfix/cleanup[4839]: 36908479D7F: message-id=<20191216084302.36908479D7F@dev1-myeg.int.myeg.com.my>
lines 1-48/1435 3%


Please help me debug this, as I am not sure where to start.
# 2  
Old 12-16-2019
Is this CentOS giving you this problem?

If so, I think this is a known bug.

Ref:
Misleading error message: Job for httpd.service failed because a configured resource limit was exceeded . Issue #2716 . systemd/systemd . GitHub

EDIT: Or rather the error message doesn't mean what is says!!! It's telling lies!!!
Last edited by hicksd8; 12-16-2019 at 09:40 AM..
# 3  
Old 12-16-2019
How about we start with something simple like:

Code:
apachectl configtest

What is the results of this config test?
# 4  
Old 12-16-2019
You have an error here i believe :
Code:
....
EnvironmentFile=/usr/local/apache-2.4.33/bin/httpd
.....

What i guess is going on, is that you are passing $OPTIONS as content of binary httpd file specified.
This is why you are getting limits error in log and similar.

Quote:
EnvironmentFile=
Similar to Environment= but reads the environment variables from a text file. The text file should contain new-line-separated variable assignments. Empty lines, lines without an "=" separator, or lines starting with ; or # will be ignored, which may be used for commenting. A line ending with a backslash will be concatenated with the following one, allowing multiline variable definitions. The parser strips leading and trailing whitespace from the values of assignments, unless you use double quotes (").
So either omit the mentioned variable (apache will use default) or use per specification above.

Hope that helps
Regards
Peasant.
# 5  
Old 12-19-2019
I have changed the httpd.service script like this (changed path to Environment File to default) :


Code:
[root@dev1-myeg system]# cat httpd.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=notify
EnvironmentFile=/opt/rh/httpd24/root/etc/sysconfig/httpd
#ExecStart=/usr/local/apache-2.4.33/bin/httpd $OPTIONS -DFOREGROUND
ExecStart=/usr/local/apache-2.4.33/bin/httpd -k start
ExecReload=/usr/local/apache-2.4.33/bin/httpd $OPTIONS -k graceful
ExecStop=/bin/kill -WINCH ${MAINPID}

# We want systemd to give httpd some time to finish gracefully, but still want
# it to kill httpd after TimeoutStopSec if something went wrong during the
# graceful stop. Normally, Systemd sends SIGTERM signal right after the
# ExecStop, which would kill httpd. We are sending useless SIGCONT here to give
# httpd time to finish.
KillSignal=SIGCONT
PrivateTmp=true

[Install]
WantedBy=multi-user.target
[root@dev1-myeg system]#

But I still get this error :


Code:
[root@dev1-myeg ~]# systemctl start httpd.service
Warning: httpd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for httpd.service failed because a configured resource limit was exceeded. See "systemctl status httpd.service" and "journalctl -xe" for details.

Running this command :


Code:
apachectl configtest

provides "Syntax OK".

Need some guidance here, as I am not sure where/how to start to troubleshoot.
# 6  
Old 12-19-2019
Enviroment file is not a apache configuration file, try removing that line entirely.
Every time you change a systemd unit, you need to reload the daemon
Code:
systemctl daemon-reload

Then try to start the service again.
Code:
systemctl start httpd

If the start fails again, check the status via systemctl status httpd -l
Also, look at apache error logs for hints, together with journalctl -xe

Regards
Peasant.
# 7  
Old 12-19-2019
This time I can reload httpd. However when I attempt to start it I get this error :


Code:
[root@dev1-myeg logs]# systemctl status httpd
- httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: failed (Result: timeout) since Thu 2019-12-19 15:16:38 +08; 1h 30min ago
  Process: 29026 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 29022 ExecStart=/usr/local/apache-2.4.33/bin/httpd -k start (code=exited, status=0/SUCCESS)
 Main PID: 29022 (code=exited, status=0/SUCCESS)

Dec 19 15:15:08 dev1-myeg.int.myeg.com.my systemd[1]: Starting The Apache HTTP Server...
Dec 19 15:15:08 dev1-myeg.int.myeg.com.my kill[29026]: kill: cannot find process ""
Dec 19 15:15:08 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service: control process exited, code=exited status=1
Dec 19 15:16:38 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service stop-sigterm timed out. Killing.
Dec 19 15:16:38 dev1-myeg.int.myeg.com.my systemd[1]: Failed to start The Apache HTTP Server.
Dec 19 15:16:38 dev1-myeg.int.myeg.com.my systemd[1]: Unit httpd.service entered failed state.
Dec 19 15:16:38 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service failed.
[root@dev1-myeg logs]#

This is the error logged in journalctl -xe :


Code:
Dec 19 15:14:02 dev1-myeg.int.myeg.com.my postfix/bounce[28209]: 79383479D7F: sender non-delivery notification: 89A5C47C211
Dec 19 15:14:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 79383479D7F: removed
Dec 19 15:14:02 dev1-myeg.int.myeg.com.my postfix/local[28208]: 89A5C47C211: to=<root@dev1-myeg.int.myeg.com.my>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (c
Dec 19 15:14:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 89A5C47C211: removed
Dec 19 15:14:09 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-19T15:14:09+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server
Dec 19 15:14:38 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-19T15:14:38+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http:/
Dec 19 15:14:49 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-19T15:14:49+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server
Dec 19 15:14:50 dev1-myeg.int.myeg.com.my polkitd[3254]: Registered Authentication Agent for unix-process:28967:805245139 (system bus name :1.277996 [/usr/bin/pkttyagent --notify-fd 5 --fal
Dec 19 15:14:50 dev1-myeg.int.myeg.com.my systemd[1]: Reloading.
Dec 19 15:14:51 dev1-myeg.int.myeg.com.my polkitd[3254]: Unregistered Authentication Agent for unix-process:28967:805245139 (system bus name :1.277996, object path /org/freedesktop/PolicyKi
Dec 19 15:15:01 dev1-myeg.int.myeg.com.my systemd[1]: Started Session 137606 of user root.
-- Subject: Unit session-137606.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-137606.scope has finished starting up.
--
-- The start-up result is done.
Dec 19 15:15:01 dev1-myeg.int.myeg.com.my CROND[28999]: (root) CMD (cd /usr/local/apache31-MYEG/htdocs/myeg_new_ui && php artisan get:survey > /usr/local/apache31-MYEG/logs/getsurvey-cron.l
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/pickup[25832]: 801D8479D7F: uid=0 from=<root>
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/cleanup[28206]: 801D8479D7F: message-id=<20191219071502.801D8479D7F@dev1-myeg.int.myeg.com.my>
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 801D8479D7F: from=<root@dev1-myeg.int.myeg.com.my>, size=2539, nrcpt=1 (queue active)
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/local[28208]: 801D8479D7F: to=<root@dev1-myeg.int.myeg.com.my>, orig_to=<root>, relay=local, delay=0.43, delays=0.41/0/0/0.02, dsn=5.2.2, s
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/cleanup[28206]: 889D247C211: message-id=<20191219071502.889D247C211@dev1-myeg.int.myeg.com.my>
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 889D247C211: from=<>, size=4538, nrcpt=1 (queue active)
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/bounce[28209]: 801D8479D7F: sender non-delivery notification: 889D247C211
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 801D8479D7F: removed
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/local[28208]: 889D247C211: to=<root@dev1-myeg.int.myeg.com.my>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (c
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 889D247C211: removed
Dec 19 15:15:08 dev1-myeg.int.myeg.com.my polkitd[3254]: Registered Authentication Agent for unix-process:29016:805246855 (system bus name :1.277999 [/usr/bin/pkttyagent --notify-fd 5 --fal
Dec 19 15:15:08 dev1-myeg.int.myeg.com.my systemd[1]: Starting The Apache HTTP Server...
-- Subject: Unit httpd.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has begun starting up.
Dec 19 15:15:08 dev1-myeg.int.myeg.com.my kill[29026]: kill: cannot find process ""
Dec 19 15:15:08 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service: control process exited, code=exited status=1
Dec 19 15:15:18 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-19T15:15:18+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http:/
Dec 19 15:15:29 dev1-myeg.int.myeg.com.my userhelper[29136]: pam_succeed_if(ovirt-container-list:auth): requirement "user = ovirtagent" was met by user "ovirtagent"
Dec 19 15:15:29 dev1-myeg.int.myeg.com.my userhelper[29136]: running '/usr/share/ovirt-guest-agent/container-list' with root privileges on behalf of 'ovirtagent'
Dec 19 15:15:29 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-19T15:15:29+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server
Dec 19 15:15:58 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-19T15:15:58+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http:/
Dec 19 15:16:01 dev1-myeg.int.myeg.com.my systemd[1]: Started Session 137607 of user root.
-- Subject: Unit session-137607.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-137607.scope has finished starting up.
lines 1410-1458/1515 96%


This in httpd error logs :


Code:
[Thu Dec 19 15:15:08.522704 2019] [ssl:warn] [pid 29022:tid 140066466289472] AH01906: dev1.myeg.com.my:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Thu Dec 19 15:15:08.532567 2019] [jk:warn] [pid 29022:tid 140066466289472] No JkShmFile defined in httpd.conf. Using default /usr/local/apache-2.4.33/logs/jk-runtime-status
[Thu Dec 19 15:15:08.556561 2019] [jk:error] [pid 29022] [Thu Dec 19 15:15:08 2019][29022:140066466289472] [debug] jk_shm_close::jk_shm.c (745): Closed shared memory /usr/local/apache-2.4.33/logs/jk-runtime-status.29022 childs=1
[Thu Dec 19 15:15:08.861044 2019] [ssl:warn] [pid 29025:tid 140066466289472] AH01906: dev1.myeg.com.my:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Thu Dec 19 15:15:09.104449 2019] [jk:warn] [pid 29025:tid 140066466289472] No JkShmFile defined in httpd.conf. Using default /usr/local/apache-2.4.33/logs/jk-runtime-status
[Thu Dec 19 15:15:09.142371 2019] [mpm_event:notice] [pid 29025:tid 140066466289472] AH00489: Apache/2.4.33 (Unix) OpenSSL/1.0.2o PHP/5.6.36 mod_jk/1.2.43 configured -- resuming normal operations
[Thu Dec 19 15:15:09.142464 2019] [core:notice] [pid 29025:tid 140066466289472] AH00094: Command line: '/usr/local/apache-2.4.33/bin/httpd'
(END)

This in httpd access logs :


Code:
46.229.168.161 - - [19/Dec/2019:14:04:03 +0800] "GET /robots.txt HTTP/1.1" 200 24 "-" "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)" TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256
46.229.168.163 - - [19/Dec/2019:14:04:05 +0800] "GET /career/mymotor HTTP/1.1" 200 5341 "-" "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)" TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256
45.56.78.64 - - [19/Dec/2019:14:24:15 +0800] "GET /?0628182016134805143312 HTTP/1.1" 400 226
64.233.172.106 - - [19/Dec/2019:15:03:04 +0800] "GET /eservices/ HTTP/1.1" 200 1607
64.233.172.108 - - [19/Dec/2019:15:03:04 +0800] "GET /common/img/favicon.ico HTTP/1.1" 200 32942
(END)

This in /var/log/messages :


Code:
Dec 19 15:14:50 dev1-myeg systemd: Reloading.
Dec 19 15:15:01 dev1-myeg systemd: Started Session 137606 of user root.
Dec 19 15:15:08 dev1-myeg systemd: Starting The Apache HTTP Server...
Dec 19 15:15:08 dev1-myeg kill: kill: cannot find process ""
Dec 19 15:15:08 dev1-myeg systemd: httpd.service: control process exited, code=exited status=1
Dec 19 15:15:18 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:15:18+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://13.228.178.92:9000/ap
i/plugins/org.graylog.plugins.collector/3fb08426-6b4b-4928-969f-913cb690dc3f?tags=%5B%22laravel%22%2C%22apache%22%5D: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:15:29 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:15:29+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://13.228.17
8.92:9000/api/plugins/org.graylog.plugins.collector/collectors/3fb08426-6b4b-4928-969f-913cb690dc3f: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:15:58 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:15:58+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://13.228.178.92:9000/ap
i/plugins/org.graylog.plugins.collector/3fb08426-6b4b-4928-969f-913cb690dc3f?tags=%5B%22laravel%22%2C%22apache%22%5D: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:16:01 dev1-myeg systemd: Started Session 137607 of user root.
Dec 19 15:16:09 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:16:09+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://13.228.17
8.92:9000/api/plugins/org.graylog.plugins.collector/collectors/3fb08426-6b4b-4928-969f-913cb690dc3f: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:16:38 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:16:38+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://13.228.178.92:9000/ap
i/plugins/org.graylog.plugins.collector/3fb08426-6b4b-4928-969f-913cb690dc3f?tags=%5B%22laravel%22%2C%22apache%22%5D: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:16:38 dev1-myeg systemd: httpd.service stop-sigterm timed out. Killing.
Dec 19 15:16:38 dev1-myeg systemd: Failed to start The Apache HTTP Server.
Dec 19 15:16:38 dev1-myeg systemd: Unit httpd.service entered failed state.
Dec 19 15:16:38 dev1-myeg systemd: httpd.service failed.
Dec 19 15:16:49 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:16:49+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://13.228.178.92:9000/api/plugins/org.graylog.plugins.collector/collectors/3fb08426-6b4b-4928-969f-913cb690dc3f: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:17:01 dev1-myeg systemd: Started Session 137608 of user root.
Dec 19 15:17:18 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:17:18+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://13.228.178.92:9000/api/plugins/org.graylog.plugins.collector/3fb08426-6b4b-4928-969f-913cb690dc3f?tags=%5B%22laravel%22%2C%22apache%22%5D: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:17:29 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:17:29+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://13.228.178.92:9000/api/plugins/org.graylog.plugins.collector/collectors/3fb08426-6b4b-4928-969f-913cb690dc3f: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:17:58 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:17:58+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://13.228.178.92:9000/api/plugins/org.graylog.plugins.collector/3fb08426-6b4b-4928-969f-913cb690dc3f?tags=%5B%22laravel%22%2C%22apache%22%5D: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:18:01 dev1-myeg systemd: Started Session 137609 of user root.
Dec 19 15:18:09 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:18:09+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://13.228.178.92:9000/api/plugins/org.graylog.plugins.collector/collectors/3fb08426-6b4b-4928-969f-913cb690dc3f: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:18:38 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:18:38+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://13.228.178.92:9000/ap:

I notice that somehow the graylog-collector-sidecar errors also show after the httpd failures, and also a lot of messages on "ovirt" appear together with them too. How are all of these related?


I ran the command :


Code:
# /usr/bin/graylog-collector-sidecar -debug

and these were the outputs I get :


Code:
[root@dev1-myeg collector-sidecar]# /usr/bin/graylog-collector-sidecar -debug
INFO[0000] Using collector-id: 3fb08426-6b4b-4928-969f-913cb690dc3f
INFO[0000] No node-id was configured, falling back to hostname
INFO[0000] Fetching configurations tagged by: [laravel apache]
DEBU[0000] Creating rotated log writer for: /var/log/graylog/collector-sidecar/collector_sidecar.log.%Y%m%d%H%M
DEBU[0000] Add collector backend: filebeat
INFO[0000] Starting signal distributor
DEBU[0000] [signal-processor] (seq=1) handling cmd: restart
INFO[0000] [filebeat] Starting (exec driver)
DEBU[0000] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0000] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0000] [signal-processor] (seq=1) cmd done: restart
ERRO[0001] [filebeat] Backend finished unexpectedly, trying to restart 1/3.
DEBU[0001] [signal-processor] (seq=2) handling cmd: restart
INFO[0001] [filebeat] Stopping
DEBU[0003] [filebeat] SIGHUP ignored, killing process
DEBU[0003] [filebeat] Failed to kill process os: process already finished
INFO[0003] [filebeat] Starting (exec driver)
DEBU[0003] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0003] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0003] [signal-processor] (seq=2) cmd done: restart
ERRO[0004] [filebeat] Backend finished unexpectedly, trying to restart 2/3.
DEBU[0004] [signal-processor] (seq=3) handling cmd: restart
INFO[0004] [filebeat] Stopping
DEBU[0006] [filebeat] SIGHUP ignored, killing process
DEBU[0006] [filebeat] Failed to kill process os: process already finished
INFO[0006] [filebeat] Starting (exec driver)
DEBU[0006] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0006] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0006] [signal-processor] (seq=3) cmd done: restart
ERRO[0007] [filebeat] Backend finished unexpectedly, trying to restart 3/3.
DEBU[0007] [signal-processor] (seq=4) handling cmd: restart
INFO[0007] [filebeat] Stopping
DEBU[0009] [filebeat] SIGHUP ignored, killing process
DEBU[0009] [filebeat] Failed to kill process os: process already finished
INFO[0009] [filebeat] Starting (exec driver)
DEBU[0009] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0009] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0009] [signal-processor] (seq=4) cmd done: restart
ERRO[0010] [filebeat] Unable to start collector after 3 tries, giving up!







ERRO[1800] [RequestConfiguration] Fetching configuration failed: Get http://13.228.178.92:9000/api/plugins/org.graylog.plugins.collector/3fb08426-6b4b-4928-969f-913cb690dc3f?tags=%5B%22laravel%22%2C%22apache%22%5D: dial tcp 13.228.178.92:9000: i/o timeout
ERRO[1800] [UpdateRegistration] Failed to report collector status to server: Put http://13.228.178.92:9000/api/plugins/org.graylog.plugins.collector/collectors/3fb08426-6b4b-4928-969f-913cb690dc3f: dial tcp 13.228.178.92:9000: i/o timeout

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

How to create custom options of my own in systemctl Linux?

Hello All, Greetings!! I was trying to create a custom option in systemctl like its out of the box options eg--> status, stop, start, restart, I have tried searching in google but didn't find anything related to it. Basically I would like to create a custom option which is related to... (9 Replies)
Discussion started by: RavinderSingh13
9 Replies

2. Shell Programming and Scripting

How to integrate all the systemctl commands into a shell script to verify any daemon/agent service?

Hi, Can we integrate all the systemctl command into a shell script to verify any service with all the options of systemctl if service integrate with the operating system service management tools to deliver their functionality. sudo systemctl start <service_name> sudo systemctl stop... (1 Reply)
Discussion started by: Mannu2525
1 Replies

3. UNIX for Beginners Questions & Answers

Systemctl enable failing

i started this command and got this error messages on red hat 7.0 # systemctl enable httpd.service Failed to execute operation: No such file or directory #systemctl enable mysql.service Failed to execute operation: Too many levels of symbolic links what should i do as this command... (2 Replies)
Discussion started by: DOkuwa
2 Replies

4. UNIX for Dummies Questions & Answers

How to grep a line not starting with # from a file (there are two lines starting with # and normal)?

e.g. File name: File.txt cat File.txt Result: #INBOUND_QUEUE=FAQ1 INBOUND_QUEUE=FAQ2 I want to get the value for one which is not commented out. Thanks, (3 Replies)
Discussion started by: Tanu
3 Replies

5. Red Hat

Httpd service not starting... Error "Documentroot must be a directory"

I am working with CentOS 6.5. Indeed the document root /var/www/html is a directory and it is a mount point of one file system (not NFS) I observed, if I unmount that file system, I could able to start httpd. Is it restricted in CentOS 6.X that the Documentroot can not be a mount point? ... (5 Replies)
Discussion started by: atanubanerji
5 Replies

6. Web Development

Httpd not starting

I compiled apache 2.2.15 from source and installed it. When I try to start it, its not coming up. # ./httpd -k start # echo $? 1 # ps -ef | grep httpd | grep -v grep # # ./httpd -v Server version: Apache/2.2.15 (Unix) Server built: Dec 13 2013 04:31:41When I run it in gdb I am... (4 Replies)
Discussion started by: chacko193
4 Replies

7. Slackware

I/O Error on starting firefox

I was trying to open the port for my ftp end also Ive triedto start firefox and so i received the following error: Input / Output Error. When i try to start that one in console i receive the following: No Protocol Especified No Protocol Especified Error: Cannot open display: :0,0 (1 Reply)
Discussion started by: darkpirate
1 Replies

8. AIX

Error when compile httpd 2.2.8 on AIX 5.2

Hi everybody, I tried to compile httpd-2.8.8 on AIX 5.2. After downloading the Source file I run the configure as following: >> CC=gcc ./configure --prefix=/usr/local/apache After that i run the command make, but it failed. Below are last several lines of stdout of the make command: lo... (7 Replies)
Discussion started by: aldowsary
7 Replies

9. Linux

splashy starting up error

Hi eveyone and Happy Holidays, My problem is starting splashy in fedora 8 what i have done is:- installed splashy with no errors edited grub removed rhgb and added quiet vga=791 splash set default theme with command splashy_config -s default no errors added /sbin/splashy boot in... (4 Replies)
Discussion started by: dave123
4 Replies

10. UNIX for Dummies Questions & Answers

httpd error

I installed FreeBSD6.x and apache2.x using gzip the tar files.i then gave ./configure command.It is know that by default apache2 will be created in /usr/local/apache2 but when i give ./configure command apache2 is not created and iam not able to execute apachectl start it says command not... (2 Replies)
Discussion started by: kumarangopi
2 Replies
Login or Register to Ask a Question