Issue Server shut down randomly

[adijeff]

Hi,

My server randomly shut down yesterday morning at 5:01am and I'd be grateful for some help to work out why.
CentOS 6.9 (Final)‬
Plesk 12.5.30 Update #74

Looking at the log files doesn't give me any clues. They are below.

The only cron job except cron.hourly that is close to that time is my daily backup which completed at 3:57am as normal with no errors. A snippet of var/log/cron is below.

I don't know where else to look for a reason for the shutdown. Please can anyone help?

Here is the relevant section of var/log/messages:

Mar  7 04:54:41 server proftpd[28079]: XX.XX.XX.XX (195.20.253.8[195.20.253.8]) - FTP session opened.
Mar  7 04:54:41 server proftpd[28079]: XX.XX.XX.XX (195.20.253.8[195.20.253.8]) - FTP session closed.
Mar  7 04:54:41 server xinetd[1920]: EXIT: ftp status=0 pid=28079 duration=0(sec)
Mar  7 05:01:08 server init: tty (/dev/tty1) main process (3429) killed by TERM signal
Mar  7 05:01:08 server init: tty (/dev/tty2) main process (3431) killed by TERM signal
Mar  7 05:01:08 server init: tty (/dev/tty3) main process (3433) killed by TERM signal
Mar  7 05:01:08 server init: tty (/dev/tty4) main process (3435) killed by TERM signal
Mar  7 05:01:08 server init: tty (/dev/tty5) main process (3437) killed by TERM signal
Mar  7 05:01:08 server init: tty (/dev/tty6) main process (3441) killed by TERM signal
Mar  7 05:01:28 server xinetd[1920]: START: ftp pid=28392 from=::ffff:188.166.223.112
Mar  7 05:01:28 server proftpd[28392]: processing configuration directory '/etc/proftpd.d'
Mar  7 05:01:29 server proftpd[28392]: XX.XX.XX.XX (188.166.223.112[188.166.223.112]) - FTP session opened.
Mar  7 05:01:30 server xinetd[1920]: Exiting...
Mar  7 05:01:30 server proftpd[28392]: XX.XX.XX.XX (188.166.223.112[188.166.223.112]) - FTP session closed.
Mar  7 05:01:30 server ntpd[1931]: ntpd exiting on signal 15
Mar  7 05:01:31 server init: Disconnected from system bus
Mar  7 05:01:31 server console-kit-daemon[3588]: WARNING: no sender#012
Mar  7 05:01:31 server named[2786]: received control channel command 'stop'
Mar  7 05:01:32 server named[2786]: shutting down: flushing changes
Mar  7 05:01:32 server named[2786]: stopping command channel on 127.0.0.1#953
Mar  7 05:01:32 server named[2786]: no longer listening on ::#53
Mar  7 05:01:32 server named[2786]: no longer listening on 127.0.0.1#53
Mar  7 05:01:32 server named[2786]: no longer listening on 87.106.216.148#53
Mar  7 05:01:33 server named[2786]: exiting
Mar  7 05:01:34 server auditd[1532]: The audit daemon is exiting.
Mar  7 05:01:34 server kernel: type=1305 audit(1520398894.302:3678625): audit_pid=0 old=1532 auid=4294967295 ses=4294967295 res=1
Mar  7 05:01:35 server kernel: type=1305 audit(1520398895.350:3678626): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 res=1
Mar  7 05:01:35 server kernel: Kernel logging (proc) stopped.
Mar  7 05:01:35 server rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1566" x-info="http://www.rsyslog.com"] exiting on signal 15.
Mar  7 08:27:42 server kernel: imklog 5.8.10, log source = /proc/kmsg started.
Mar  7 08:27:42 server rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1707" x-info="http://www.rsyslog.com"] start
Mar  7 08:27:42 server kernel: Initializing cgroup subsys cpuset
Mar  7 08:27:42 server kernel: Initializing cgroup subsys cpu
Mar  7 08:27:42 server kernel: Linux version 2.6.32-696.13.2.el6.x86_64 ([email protected]) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC) ) #1 SMP Thu Oct 5 21:22:16 UTC 2017
Mar  7 08:27:42 server kernel: Command line: ro root=/dev/md1 console=tty0 console=ttyS0,57600 crashkernel=auto SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=us

Here is var/log/httpd/error.log:

[Wed Mar 07 04:59:02 2018] [error] [client 216.244.66.241] File does not exist: /var/www/vhosts/default/htdocs/forums
[Wed Mar 07 04:59:04 2018] [error] [client 216.244.66.241] File does not exist: /var/www/vhosts/default/htdocs/forums
[Wed Mar 07 04:59:06 2018] [error] [client 216.244.66.241] File does not exist: /var/www/vhosts/default/htdocs/forums
[Wed Mar 07 05:01:20 2018] [notice] caught SIGTERM, shutting down
[Wed Mar 07 08:28:01 2018] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Mar 07 08:28:03 2018] [warn] RSA server certificate CommonName (CN) `Parallels Panel' does NOT match server name!?
[Wed Mar 07 08:28:03 2018] [warn] RSA server certificate CommonName (CN) `Parallels Panel' does NOT match server name!?

Here is var/log/httpd/access.log:

127.0.0.1 - - [07/Mar/2018:04:54:59 +0000] "GET / HTTP/1.1" 200 7153 "-" "-"
216.244.66.241 - - [07/Mar/2018:04:59:02 +0000] "GET /forums/antabuse-prescription-costs-there-generic-equivalent-antabuse-wmpfh HTTP/1.0" 404 272 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])"
216.244.66.241 - - [07/Mar/2018:04:59:04 +0000] "GET /forums/buy-aciclovir-cheap-uk-legal-buy-aciclovir-online-z191x HTTP/1.0" 404 260 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])"
216.244.66.241 - - [07/Mar/2018:04:59:06 +0000] "GET /forums/buy-atarax-tablets-online-atarax-prescription-cost-azmmw HTTP/1.0" 404 261 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])"
127.0.0.1 - - [07/Mar/2018:04:59:59 +0000] "GET / HTTP/1.1" 200 7153 "-" "-"
38.77.208.35 - - [07/Mar/2018:08:32:46 +0000] "GET /robots.txt HTTP/1.0" 404 208 "-" "Dispatch/0.11.3"
38.94.188.35 - - [07/Mar/2018:08:32:50 +0000] "GET / HTTP/1.0" 200 7153 "-" "Dispatch/0.11.3"

Here is var/log/cron:

Mar  7 05:00:01 server CROND[28125]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar  7 05:00:01 server CROND[28127]: (root) CMD (/usr/local/psa/admin/bin/php -dauto_prepend_file=sdk.php '/usr/local/psa/admin/plib/modules/plesk-mobile/scripts/push_worker.php')
Mar  7 05:00:01 server CROND[28129]: (root) CMD (wget -q -O /dev/null "http://pshmn.com/YYYYYYY")
Mar  7 05:01:01 server CROND[28149]: (root) CMD (run-parts /etc/cron.hourly)
Mar  7 05:01:01 server run-parts(/etc/cron.hourly)[28149]: starting 0anacron
Mar  7 05:01:01 server run-parts(/etc/cron.hourly)[28158]: finished 0anacron
Mar  7 05:01:01 server run-parts(/etc/cron.hourly)[28149]: starting plesk-php-cleanuper
Mar  7 05:01:02 server run-parts(/etc/cron.hourly)[28176]: finished plesk-php-cleanuper
Mar  7 05:01:30 server crond[3186]: (CRON) INFO (Shutting down)
Mar  7 08:28:17 server crond[3329]: (CRON) STARTUP (1.4.4)

Here is var/log/secure:

Mar  7 05:01:09 server sshd[28179]: Failed password for root from 121.18.238.39 port 45358 ssh2
Mar  7 05:01:11 server sshd[28179]: Failed password for root from 121.18.238.39 port 45358 ssh2
Mar  7 05:01:13 server sshd[28179]: Failed password for root from 121.18.238.39 port 45358 ssh2
Mar  7 05:01:13 server sshd[28180]: Received disconnect from 121.18.238.39: 11:
Mar  7 05:01:13 server sshd[28179]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.39  user=root
Mar  7 05:01:15 server sshd[28250]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.39  user=root
Mar  7 05:01:17 server sshd[28250]: Failed password for root from 121.18.238.39 port 53324 ssh2
Mar  7 05:01:19 server sshd[28250]: Failed password for root from 121.18.238.39 port 53324 ssh2
Mar  7 05:01:21 server sshd[28250]: Failed password for root from 121.18.238.39 port 53324 ssh2
Mar  7 05:01:21 server sshd[28251]: Received disconnect from 121.18.238.39: 11:
Mar  7 05:01:21 server sshd[28250]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.39  user=root
Mar  7 05:01:23 server sshd[28298]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.39  user=root
Mar  7 05:01:23 server sshd[1909]: Received signal 15; terminating.
Mar  7 05:01:30 server proftpd[28392]: XX.XX.XX.XX (188.166.223.112[188.166.223.112]) - USER server: no such user found from 188.166.223.112 [188.166.223.112] to XX.XX.XX.XX:21
Mar  7 08:27:43 server runuser: pam_unix(runuser:session): session opened for user root by (uid=0)

 

[Émerson Felinto]

This problem does not seem to be related to the basic configuration of Plesk. The provider of your dedicated / vps may very well have shut down your server by mistake ...
I recommend asking your provider if they have had any problems recently that might have caused this.

I recommend you read this thread, maybe I can help you figure out where the problem is: How to find out from the logs what caused system shutdown?

 

[adijeff]

This problem does not seem to be related to the basic configuration of Plesk. The provider of your dedicated / vps may very well have shut down your server by mistake ...
I recommend asking your provider if they have had any problems recently that might have caused this.

Thanks Emerson. I already asked them, and they said:

As for the original reason to the server going down, it will be a job for you system administrator to check the log files.

As for your link How to find out from the logs what caused system shutdown? I still don't know. When I use

last -x shutdown reboot

I get this which doesn't really help:

[root@server ~]# last -x shutdown reboot
reboot   system boot  2.6.32-696.13.2. Wed Mar  7 08:27 - 11:30 (1+03:02)
shutdown system down  2.6.32-696.13.2. Wed Mar  7 05:01 - 08:27  (03:25)
reboot   system boot  2.6.32-696.13.2. Wed Jan 31 22:50 - 05:01 (34+06:10)

 

[Émerson Felinto]

Hello,
I would really like to help you but I understand a lot more about Plesk than how the Linux works.

(Log files will scare you ..)

I really hope someone can give you a useful answer, but remember that an unexpected shutdown for example ... will not generate any useful log. So it really is very complicated to know the cause of your problem.

I recommend creating a topic in ServerFault >>> Server Fault

There they will probably suggest a series of log files for you to analyze and try to figure out the cause of the problem.

PS: It is for these and other reasons that you must have a professional service that manages the logs for you. I currently use Papertrail and am very pleased. The setup is a little nebulous, but it has already gotten me out of many problems!

 

[MarkM]

SO it looks like you have a clean reboot at the time so it's possible it was caused by a kernel panic. You have an old kernel. The latest for CentOS 6.9 appears to be 2.6.32-696.20.1 - I'd start by update the kernel and then see if it reoccurs.

# yum -y update kernel

Followed by a reboot to boot into the new kernel.

 

[adijeff]

Thanks Mark. I ran # yum -y update kernel:

[root@server ~]# yum -y update kernel
Loaded plugins: fastestmirror
Setting up Update Process
Loading mirror speeds from cached hostfile
 * rpmforge: mirror1.hs-esslingen.de
base                                                                             | 3.7 kB     00:00
extras                                                                           | 3.4 kB     00:00
google-mod-pagespeed                                                             |  951 B     00:00
mod-pagespeed                                                                    |  951 B     00:00
mysql-connectors-community                                                       | 2.5 kB     00:00
mysql-tools-community                                                            | 2.5 kB     00:00
mysql56-community                                                                | 2.5 kB     00:00
nodesource                                                                       | 2.5 kB     00:00
plesk-php-5.4                                                                    | 2.9 kB     00:00
plesk-php-5.6                                                                    | 2.9 kB     00:00
plesk-php-7.0                                                                    | 2.9 kB     00:00
rpmforge                                                                         | 1.9 kB     00:00
updates                                                                          | 3.4 kB     00:00
No Packages marked for Update
[root@server ~]#

A reboot again failed to bring the server back up. The host admins have managed to get the server to boot back up, and uname -r shows no change in the kernel:

[root@server ~]# uname -r
2.6.32-696.13.2.el6.x86_64
[root@server ~]#

I have asked the admins what caused the reboot failure, and they're looking into it.

 

[MarkM]

Is this a OpenVZ VPS? There should be a kernel update unless in the OpenVZ situation where it's utilizing the host's kernel.

 

[adijeff]

No it's not OpenVZ, it's a dedicated server.

 

[MarkM]

Can you check /etc/yum.conf and verify you don't have it listed as an exclude

 

[adijeff]

/etc/yum.conf is there with 644 permissions:

[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=5
bugtracker_url=http://bugs.centos.org/set_project.php?project_id=19&ref=http://bugs.centos.org/bug_report_page.php?category=yum
distroverpkg=centos-release

Sorry, I don't know what you mean by having it listed as an exclude.

 

[MarkM]

Just noticed I forgot to ask - did you see anything in /var/log/dmesg for that time?

 

[IgorG]

This may be caused by a hardware failure. Scanning RAM with memtester, for example, would be a good idea.

 

[adijeff]

/var/log/dmesg doesn't have any timestamps on it, so I can't be sure.

The server admins have said they will investigate the issue next Wednesday, so any hardware failures should be known then.

I will post any results here. Thanks for all your help.