Issue dbus failure - Looping too fast. Throttling execution a little.

[Pan_Duke]

Suddenly this Centos 7.2/Plesk 12.5 dedicated server restarted and never booted again. most services fail. i don't have ssh so i can only go through rescue system (Debian) and check the logs.
i am in the darkness here. Could this be Plesk related. I didn't make any changes whatsoever for a long time but plesk keeps updating regularly.
Is there any plesk logs i can check in order to find what the problem could be?
As far as my search goes it seems that dbus fails and everything else follows. the whole things ends in a loop

Jan 17 02:09:58 dedicated-server1 chronyd[769]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH)
Jan 17 02:09:58 dedicated-server1 systemd: Starting The PHP 5.5.38 FastCGI Process Manager...
Jan 17 02:09:58 dedicated-server1 systemd: Starting Dump dmesg to /var/log/dmesg...
Jan 17 02:09:58 dedicated-server1 systemd: Starting Login Service...
Jan 17 02:09:58 dedicated-server1 systemd: Starting The Apache HTTP Server...
Jan 17 02:09:58 dedicated-server1 systemd: Starting Startup script for Plesk control panel server...
Jan 17 02:09:58 dedicated-server1 systemd: Starting Dynamic System Tuning Daemon...
Jan 17 02:09:58 dedicated-server1 systemd: Starting MagicSpam AntiSpam service daemon...
Jan 17 02:09:58 dedicated-server1 systemd: Starting ACPI Event Daemon...
Jan 17 02:09:58 dedicated-server1 systemd: Started Dovecot IMAP/POP3 email server.
Jan 17 02:09:58 dedicated-server1 systemd: Starting Dovecot IMAP/POP3 email server...
Jan 17 02:09:58 dedicated-server1 systemd: Started D-Bus System Message Bus.
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Could not get password database information for UID of current process: User "???" unknown or no memory to allocate password entry
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Unknown username "root" in message bus configuration file
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Could not get password database information for UID of current process: User "???" unknown or no memory to allocate password entry
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Unknown username "root" in message bus configuration file
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Could not get password database information for UID of current process: User "???" unknown or no memory to allocate password entry
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Unknown username "root" in message bus configuration file
Jan 17 02:09:58 dedicated-server1 chronyd[769]: Fatal error : getpwnam(chrony) failed
Jan 17 02:09:58 dedicated-server1 chronyd: getpwnam(chrony) failed
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Could not get password database information for UID of current process: User "???" unknown or no memory to allocate password entry
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Unknown username "root" in message bus configuration file
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Could not get password database information for UID of current process: User "???" unknown or no memory to allocate password entry
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Unknown username "root" in message bus configuration file
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Could not get password database information for UID of current process: User "???" unknown or no memory to allocate password entry
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Unknown username "root" in message bus configuration file
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Could not get password database information for UID of current process: User "???" unknown or no memory to allocate password entry
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Unknown username "root" in message bus configuration file
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Could not get password database information for UID of current process: User "???" unknown or no memory to allocate password entry
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Unknown username "root" in message bus configuration file
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Could not get password database information for UID of current process: User "???" unknown or no memory to allocate password entry
Jan 17 02:09:58 dedicated-server1 dbus-daemon: Failed to start message bus: Could not get UID and GID for username "dbus"
Jan 17 02:09:58 dedicated-server1 magicspam-daemon: Starting magicspam-daemon ...groupadd: group 'magicspam' already exists
Jan 17 02:09:58 dedicated-server1 magicspam-daemon: Failed to initialize user magicspam
Jan 17 02:09:58 dedicated-server1 smartd[764]: smartd 6.2 2013-07-26 r3841 [x86_64-linux-3.10.0-327.28.3.el7.x86_64] (local build)
Jan 17 02:09:58 dedicated-server1 smartd[764]: Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
...............
...............
Jan 17 02:10:48 dedicated-server1 systemd: systemd-logind.service: main process exited, code=exited, status=1/FAILURE
Jan 17 02:10:48 dedicated-server1 systemd: Failed to start Login Service.
Jan 17 02:10:48 dedicated-server1 systemd: Unit systemd-logind.service entered failed state.
Jan 17 02:10:48 dedicated-server1 systemd: systemd-logind.service failed.
Jan 17 02:10:48 dedicated-server1 systemd: httpd.service: main process exited, code=exited, status=1/FAILURE
Jan 17 02:10:48 dedicated-server1 systemd: sw-cp-server.service: control process exited, code=exited status=1
Jan 17 02:10:48 dedicated-server1 systemd: Failed to start Startup script for Plesk control panel server.
Jan 17 02:10:48 dedicated-server1 systemd: Dependency failed for Plesk Panel.
Jan 17 02:10:48 dedicated-server1 systemd: Job psa.service/start failed with result 'dependency'.
Jan 17 02:10:48 dedicated-server1 systemd: Unit sw-cp-server.service entered failed state.
Jan 17 02:10:48 dedicated-server1 systemd: sw-cp-server.service failed.
Jan 17 02:10:48 dedicated-server1 systemd: tuned.service: main process exited, code=exited, status=1/FAILURE
Jan 17 02:10:48 dedicated-server1 systemd: Failed to start Dynamic System Tuning Daemon.
Jan 17 02:10:48 dedicated-server1 systemd: Unit tuned.service entered failed state.
Jan 17 02:10:48 dedicated-server1 systemd: tuned.service failed.
Jan 17 02:10:48 dedicated-server1 systemd: magicspam-daemon.service: control process exited, code=exited status=1
Jan 17 02:10:48 dedicated-server1 kill: kill: cannot find process ""
Jan 17 02:10:48 dedicated-server1 systemd: Failed to start MagicSpam AntiSpam service daemon.
Jan 17 02:10:48 dedicated-server1 xinetd[1071]: xinetd Version 2.3.15 started with libwrap loadavg labeled-networking options compiled in.
Jan 17 02:10:48 dedicated-server1 systemd: Unit magicspam-daemon.service entered failed state.
Jan 17 02:10:48 dedicated-server1 xinetd[1071]: Started working: 0 available services
Jan 17 02:10:48 dedicated-server1 systemd: magicspam-daemon.service failed.
.................
.............
.....
Jan 17 02:10:48 dedicated-server1 nginx: nginx: [emerg] getpwnam("nginx") failed
Jan 17 02:10:48 dedicated-server1 nginx: nginx: configuration file /etc/nginx/nginx.conf test failed
Jan 17 02:10:48 dedicated-server1 systemd: nginx.service: control process exited, code=exited status=1
Jan 17 02:10:48 dedicated-server1 systemd: Failed to start Startup script for nginx service.
Jan 17 02:10:48 dedicated-server1 systemd: Unit nginx.service entered failed state.
Jan 17 02:10:48 dedicated-server1 systemd: nginx.service failed.
Jan 17 02:10:48 dedicated-server1 systemd: Looping too fast. Throttling execution a little.
...............
...............
Jan 17 02:10:49 dedicated-server1 systemd: Failed to start The PHP 7.0.27 FastCGI Process Manager.
Jan 17 02:10:49 dedicated-server1 systemd: Unit plesk-php70-fpm.service entered failed state.
Jan 17 02:10:49 dedicated-server1 systemd: plesk-php70-fpm.service failed.
Jan 17 02:10:49 dedicated-server1 systemd: sw-engine.service: main process exited, code=exited, status=78/n/a
Jan 17 02:10:49 dedicated-server1 systemd: Failed to start Startup script for Panel sw-engine.
Jan 17 02:10:49 dedicated-server1 systemd: Unit sw-engine.service entered failed state.
Jan 17 02:10:49 dedicated-server1 systemd: sw-engine.service failed.
Jan 17 02:10:49 dedicated-server1 systemd: plesk-php54-fpm.service: main process exited, code=exited, status=78/n/a
Jan 17 02:10:49 dedicated-server1 systemd: Failed to start The PHP 5.4.45 FastCGI Process Manager.
Jan 17 02:10:49 dedicated-server1 systemd: Unit plesk-php54-fpm.service entered failed state.
Jan 17 02:10:49 dedicated-server1 systemd: plesk-php54-fpm.service failed.
Jan 17 02:10:49 dedicated-server1 systemd: plesk-php56-fpm.service: main process exited, code=exited, status=78/n/a
Jan 17 02:10:49 dedicated-server1 systemd: Failed to start The PHP 5.6.33 FastCGI Process Manager.
Jan 17 02:10:49 dedicated-server1 systemd: Unit plesk-php56-fpm.service entered failed state.
Jan 17 02:10:49 dedicated-server1 systemd: plesk-php56-fpm.service failed.
Jan 17 02:10:49 dedicated-server1 systemd: Started Spamassassin daemon.
Jan 17 02:10:49 dedicated-server1 systemd: postfix.service: control process exited, code=exited status=1
Jan 17 02:10:49 dedicated-server1 systemd: Failed to start Postfix Mail Transport Agent.
Jan 17 02:10:49 dedicated-server1 systemd: Unit postfix.service entered failed state.
Jan 17 02:10:49 dedicated-server1 systemd: postfix.service failed.
Jan 17 02:10:49 dedicated-server1 systemd: [email protected] holdoff time over, scheduling restart.
Jan 17 02:10:49 dedicated-server1 systemd: fail2ban.service holdoff time over, scheduling restart.
Jan 17 02:10:49 dedicated-server1 systemd: Looping too fast. Throttling execution a little.
Jan 17 02:10:50 dedicated-server1 systemd: Looping too fast. Throttling execution a little.

Thanks in advance

 

[IgorG]

First of all try to repair your Plesk with command

# plesk repair installation

 

[Pan_Duke]

ok this was nasty... i couldn't paste all the logs in the first post but there it was. could't find root...
etc/passwd file was emptied...
restore it back,then plesk repair all -y, then full upgrade of ....everything(centos, plesk , all website CMS etc) and repair again.
now it is back and running fine.
It seems like an old Joomla was compromised... probably. But i still wonder is it possible to reach passwd file by compromising a website?

 

[Pan_Duke]

and just few minutes ago /var /etc /lib /usr and other folders popped up in the home directory of a few websites... is this a plesk problem?
i run
plesk repair installation
plesk repair all -y

but nothing changed. the folders are there. inside etc there passwd, group and resolv.conf files

this doesn't look good at all. Any suggestions?

 

[Bitpalast]

When open_basedir (find it in a subscription's PHP settings) does not limit PHP access to the subscription, it is possible to access and manipulate every other file on a system, including /etc/passwd.

When your server's temporary directory /tmp allows the execution of files, this can also lead to access through a subscription.

When your subscription's SSH settings are not set to "chrooted", full access to your system is possible through the subscription's SSH login.

The /var, /etc, /lib, /usr folders exist in a subscription's base directory, when SSH access is granted, regardless whether it is chrooted or not. This is not an error.

 

[Pan_Duke]

Peter thank you so much for the information. i knew nothing about all that....

1. i almost every time use the default open_dir setting. does this limits PHP access?
2. just checked and server's tmp is set to 777... is this default? would 755 be more wise or would the server have problems operating?
3. Checking the websites affected i can see at Web Hosting Access that Access to the server over SSH is set to Forbitten. Still /var, /etc, /lib, /usr. Could this be an anomaly after the upgrade from 12.5 to 17?

 

[Bitpalast]

1) The default open_basedir setting is good enough. It allows access to the subscription's base directory and lower, but not higher. The correct minimum setting is:
{WEBSPACEROOT}{/}{:}{TMP}{/}
For ownCloud and Nextcloud installations you should give access to /dev/urandom, too:
{WEBSPACEROOT}{/}{:}{TMP}{/}{:}/dev/urandom

2) Please see Making /tmp non-executable for a better explanation. It's too much to explain all that here. Basically it must not be possible to execute any file from the /tmp location. If the "noexec" and "nosuid" are not set for the /tmp partition, sooner or later you'll be in trouble, because some hacks allow access to the root level through an execution from /tmp if you are not protecting this gap.

3) No, the directories should not exist if no SSH access is granted to a subscription. I do not know why they exist in your case, but it does not look correct and it is not created by Plesk in this case.

 

[Pan_Duke]

1. ok this is clear
2. I will look in to this asap. this is serious, i wonder why tmp has full rights by default. The article is for Debian, is it the same for Centos?
3. Now you got me worried. So i made a very standard move. i backed up, deleted all subdomains, changed "hosting type" to "No Website Hosting", then back to "website hosting" so that i don't lose emails but recreate all structure and settings, and tried to restore databases and folders one by one from the backup. i got all kind of errors, ending downloading the backup and uploading one by one the parts in order to restore... Finally i ended with php70-fpm down. Had to manually delete the subdomain conf from /opt/plesk/php/7.0/etc/php-fpm.d and now it looks normal....
I am seriously thinking of reinstalling the server from scratch, we all know that this is a big pain but my major concern is if the backups are going to work....
So one last thing, is there any other command to run (or anything else...) in order to "save" this freekin installation?

Again, thanks a million Peter i learned many things today

 

[Bitpalast]

It is basically the same for CentOS and other flavors. There is no "default" for /tmp, it is simply created with the typical default of partitions, because the OS does not know that /tmp is /tmp. It is left up to admins to enable or disable exec permissions.

 

[Pan_Duke]

and... /var, /etc, /lib, /usr and all other folders popped back in!
could this be a server recurring event that doesn't work as expected?
this is not happening to all subscriptions.... only to 3....
on the other hand i don't see anything suspicious going on (at least for now)