Issue Plesk webserver inaccessible

[jola]

Server operating system version

Centos 7.9 ELS

Plesk version and microupdate number

Plesk Obsidian 18.0.66.1

All of a sudden my Plesk webserver is inaccessible. None of the websites hosted on it are working, email IMAP is also not responding, and I can not reach my :8443 plesk login page.

This is Plesk Obsidian (latest version) running on a Centos 7.9 ELS server. I can login via ssh. When I check the logs I can see a problem in /var/logs/plesk/panel.log:

[2024-12-23 01:41:56.446] 16210:6768b1d46cca3 ERR [panel] DB query failed: SQLSTATE[HY000] [2002] No such file or directory:
0: /usr/local/psa/admin/plib/Db/Adapter/Pdo/Mysql.php:79
Db_Adapter_Pdo_Mysql->query(string 'SET sql_mode = ''')
1: /usr/local/psa/admin/plib/CommonPanel/Application/Abstract.php:113
CommonPanel_Application_Abstract::initDbAdapter(boolean true)
2: /usr/local/psa/admin/plib/db.php:10
db_connect()
3: /usr/local/psa/admin/plib/modules/watchdog/wdlib.php:1092
wd__db_connect()
4: /usr/local/psa/admin/sbin/modules/watchdog/killwdcollect:5

I did find a few messages about this that said it was due to a problem with the mariadb database, but my mariadb service seems to be running well.

If I run plesk repair all -n I do not get any error messages and all checks give "OK".

Any clues what I should do to get my server online again?

 

[pleskpanel]

What is the output of

# service mariadb status

 

[jola]

It seems to be up and running:

â mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2024-12-23 03:18:52 CET; 25min ago
Process: 1116 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=0/SUCCESS)
Process: 1035 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Main PID: 1115 (mysqld_safe)
Tasks: 24
Memory: 1.5G
CGroup: /system.slice/mariadb.service
ââ1115 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
ââ1361 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql...

 

[pleskpanel]

Just based on that entry it was restarted 25 minutes ago so that indicates a crash/restart or possible OOM killer if you didn't kick off any updates.

Have you checked your /var/log/mariadb/ log as well as the /var/log/messages log file for activity around that time?

If Plesk and the websites are still down, have you tried restarting the server?

Before restarting it might be a good idea to take a quick copy of your databases:

# service mariadb stop
# cp -v -a /var/lib/mysql/ /var/lib/mysql_backup
# service mariadb start

Then restart and see if everything comes back normally (give the restart a couple of minutes once services start to load to ensure that the database server starts up). If the database crashes/restarts on its own then it's either a corrupt table or OOM killer.

If everything is still down, check the logs and if it's still problematic, reach out to your hosting provider if they provide Plesk support (and are quick) or contact Plesk's direct support service.

 

[WebHostingAce]

Even the MySQL is currently running, it is in the 'Main PID: 1115 (mysqld_safe)' mode.

Please post your my.cnf file.

#cat /etc/my.cnf

 

[jola]

Even the MySQL is currently running, it is in the 'Main PID: 1115 (mysqld_safe)' mode.

Please post your my.cnf file.

#cat /etc/my.cnf

Here comes my.cnf, not much interesting there:

# cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in Redirect Notice

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

 

[jola]

Just based on that entry it was restarted 25 minutes ago so that indicates a crash/restart or possible OOM killer if you didn't kick off any updates.

Yes, I've been testing several things and have rebooted several time, hence the short uptime. "plesk repair mysql" and "plesk repair mysql --connection" does not show any problems with the database.

 

[WebHostingAce]

Please post the complete output of the,

#service mariadb status

 

[jola]

Please post the complete output of the,

#service mariadb status

Here comes the output, where I removed my real domain name:

Redirecting to /bin/systemctl status mariadb.service
â mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2024-12-23 04:17:04 CET; 35min ago
Process: 1238 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=0/SUCCESS)
Process: 1116 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Main PID: 1237 (mysqld_safe)
Tasks: 23
Memory: 145.7M
CGroup: /system.slice/mariadb.service
ââ1237 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
ââ1593 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock

Dec 23 04:16:45 domain-name-removed.com systemd[1]: Starting MariaDB database server...
Dec 23 04:16:47 domain-name-removed.com mariadb-prepare-db-dir[1116]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Dec 23 04:16:47 domain-name-removed.com mariadb-prepare-db-dir[1116]: If this is not the case, make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.
Dec 23 04:16:48 domain-name-removed.com mysqld_safe[1237]: 241223 04:16:48 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Dec 23 04:16:48 domain-name-removed.com mysqld_safe[1237]: 241223 04:16:48 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Dec 23 04:17:04 domain-name-removed.com systemd[1]: Started MariaDB database server.

 

[WebHostingAce]

Please post the output of,

#cat /etc/my.cnf.d

I'm not sure why the MySQL is starting as mysqld_safe.

Also the,

#cat /var/log/mariadb/mariadb.log

 

[WebHostingAce]

Also,

#df -h

#free -h

 

[jola]

Please post the output of,

#cat /etc/my.cnf.d

Here comes the content of my files in my.cnf.d:

cat /etc/my.cnf.d/*
#
# These two groups are read by the client library
# Use it for options that affect all clients, but not the server
#


[client]

# This group is not read by mysql client library,
# If you use the same .cnf file for MySQL and MariaDB,
# use it for MariaDB-only client options
[client-mariadb]

#
# These groups are read by MariaDB command-line tools
# Use it for options that affect only one utility
#

[mysql]

[mysql_upgrade]

[mysqladmin]

[mysqlbinlog]

[mysqlcheck]

[mysqldump]

[mysqlimport]

[mysqlshow]

[mysqlslap]

# Configure utf8mb4 default character set for mysqldump
# If you don't need it, please comment out the lines below
[mysqldump]
default-character-set=utf8mb4
#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
#
# See the examples of server my.cnf files in /usr/share/mysql/
#

# this is read by the standalone daemon and embedded servers
[server]

# this is only for the mysqld standalone daemon
[mysqld]

# this is only for embedded server
[embedded]

# This group is only read by MariaDB-5.5 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mysqld-5.5]

# These two groups are only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]

[mariadb-5.5]

#cat /var/log/mariadb/mariadb.log

Here comes the last log-lines after a reboot:

# cat /var/log/mariadb/mariadb.log | tail -16
241223 06:55:55 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
241223 6:55:55 [Note] /usr/libexec/mysqld (mysqld 5.5.68-MariaDB) starting as process 1530 ...
241223 6:55:55 InnoDB: The InnoDB memory heap is disabled
241223 6:55:55 InnoDB: Mutexes and rw_locks use GCC atomic builtins
241223 6:55:55 InnoDB: Compressed tables use zlib 1.2.7
241223 6:55:55 InnoDB: Using Linux native AIO
241223 6:55:55 InnoDB: Initializing buffer pool, size = 128.0M
241223 6:55:55 InnoDB: Completed initialization of buffer pool
241223 6:55:55 InnoDB: highest supported file format is Barracuda.
241223 6:55:55 InnoDB: Waiting for the background threads to start
241223 6:55:56 Percona XtraDB (Open Source Database Software Support & Services | Percona) 5.5.61-MariaDB-38.13 started; log sequence number 19385319534
241223 6:55:56 [Note] Plugin 'FEEDBACK' is disabled.
241223 6:55:56 [Note] Server socket created on IP: '0.0.0.0'.
241223 6:55:56 [Note] Event Scheduler: Loaded 0 events
241223 6:55:56 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.68-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server

 

[jola]

Also,

#df -h

I changed an IP number and a domain-related name part, but here is the output excluding that:

# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 15G 0 15G 0% /dev
tmpfs 15G 0 15G 0% /dev/shm
tmpfs 15G 8.8M 15G 1% /run
tmpfs 15G 0 15G 0% /sys/fs/cgroup
/dev/mapper/centos_removed-root 50G 11G 40G 22% /
/dev/vda1 497M 165M 333M 34% /boot
/dev/mapper/centos_removed-var 236G 171G 65G 73% /var
1.2.3.4:/backup 5.7T 230G 5.5T 4% /backup2
tmpfs 2.9G 0 2.9G 0% /run/user/10001
tmpfs 2.9G 0 2.9G 0% /run/user/0
tmpfs 2.9G 0 2.9G 0% /run/user/998

#free -h

Here is that:

# free -h
total used free shared buff/cache available
Mem: 28G 1.2G 26G 24M 1.1G 26G
Swap: 14G 0B 14G

For your information. This is a virtual kvm based server running Centos 7.9 ELS with the latest Plesk Obsidian.

 

[WebHostingAce]

Have to see more of #cat /var/log/mariadb/mariadb.log at least last 50 lines

I'm wondering what is causing the MariaDB to start as mysqld_safe

 

[WebHostingAce]

Or you can, rename the current log file,

#mv /var/log/mariadb/mariadb.log /var/log/mariadb/mariadb.log1

Then,

#service mariadb restart

Then see the new entries in the /var/log/mariadb/mariadb.log, this should probably explain the issue.

 

[jola]

Have to see more of #cat /var/log/mariadb/mariadb.log at least last 50 lines

I'm wondering what is causing the MariaDB to start as mysqld_safe

Here comes the last 100 lines:

#cat /var/log/mariadb/mariadb.log | tail -100
241223 4:17:00 InnoDB: Waiting for the background threads to start
241223 4:17:01 Percona XtraDB (Open Source Database Software Support & Services | Percona) 5.5.61-MariaDB-38.13 started; log sequence number 19384659406
241223 4:17:01 [Note] Plugin 'FEEDBACK' is disabled.
241223 4:17:02 [Note] Server socket created on IP: '0.0.0.0'.
241223 4:17:04 [Note] Event Scheduler: Loaded 0 events
241223 4:17:04 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.68-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
241223 6:47:29 [Note] /usr/libexec/mysqld: Normal shutdown
241223 6:47:29 [Note] Event Scheduler: Purging the queue. 0 events
241223 6:47:29 InnoDB: Starting shutdown...
241223 6:47:30 InnoDB: Shutdown completed; log sequence number 19385319534
241223 6:47:30 [Note] /usr/libexec/mysqld: Shutdown complete

241223 06:47:31 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
241223 06:49:04 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
241223 6:49:05 [Note] /usr/libexec/mysqld (mysqld 5.5.68-MariaDB) starting as process 1349 ...
241223 6:49:07 InnoDB: The InnoDB memory heap is disabled
241223 6:49:07 InnoDB: Mutexes and rw_locks use GCC atomic builtins
241223 6:49:07 InnoDB: Compressed tables use zlib 1.2.7
241223 6:49:07 InnoDB: Using Linux native AIO
241223 6:49:07 InnoDB: Initializing buffer pool, size = 128.0M
241223 6:49:07 InnoDB: Completed initialization of buffer pool
241223 6:49:07 InnoDB: highest supported file format is Barracuda.
241223 6:49:14 InnoDB: Waiting for the background threads to start
241223 6:49:15 Percona XtraDB (Open Source Database Software Support & Services | Percona) 5.5.61-MariaDB-38.13 started; log sequence number 19385319534
241223 6:49:15 [Note] Plugin 'FEEDBACK' is disabled.
241223 6:49:16 [Note] Server socket created on IP: '0.0.0.0'.
241223 6:49:16 [Note] Event Scheduler: Loaded 0 events
241223 6:49:16 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.68-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
241223 6:50:42 [Note] /usr/libexec/mysqld: Normal shutdown
241223 6:50:42 [Note] Event Scheduler: Purging the queue. 0 events
241223 6:50:42 InnoDB: Starting shutdown...
241223 6:50:43 InnoDB: Waiting for 1 pages to be flushed
241223 6:50:45 InnoDB: Shutdown completed; log sequence number 19385320149
241223 6:50:45 [Note] /usr/libexec/mysqld: Shutdown complete

241223 06:50:45 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
241223 06:51:46 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
241223 6:51:48 [Note] /usr/libexec/mysqld (mysqld 5.5.68-MariaDB) starting as process 1527 ...
241223 6:51:49 InnoDB: The InnoDB memory heap is disabled
241223 6:51:49 InnoDB: Mutexes and rw_locks use GCC atomic builtins
241223 6:51:49 InnoDB: Compressed tables use zlib 1.2.7
241223 6:51:49 InnoDB: Using Linux native AIO
241223 6:51:50 InnoDB: Initializing buffer pool, size = 128.0M
241223 6:51:50 InnoDB: Completed initialization of buffer pool
241223 6:51:50 InnoDB: highest supported file format is Barracuda.
241223 6:51:56 InnoDB: Waiting for the background threads to start
241223 6:51:57 Percona XtraDB (Open Source Database Software Support & Services | Percona) 5.5.61-MariaDB-38.13 started; log sequence number 19385320149
241223 6:51:57 [Note] Plugin 'FEEDBACK' is disabled.
241223 6:51:57 [Note] Server socket created on IP: '0.0.0.0'.
241223 6:51:58 [Note] Event Scheduler: Loaded 0 events
241223 6:51:58 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.68-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
241223 6:52:09 [Note] /usr/libexec/mysqld: Normal shutdown
241223 6:52:09 [Note] Event Scheduler: Purging the queue. 0 events
241223 6:52:12 InnoDB: Starting shutdown...
241223 6:52:16 InnoDB: Shutdown completed; log sequence number 19385320464
241223 6:52:16 [Note] /usr/libexec/mysqld: Shutdown complete

241223 06:52:16 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
241223 06:53:37 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
241223 6:53:39 [Note] /usr/libexec/mysqld (mysqld 5.5.68-MariaDB) starting as process 1627 ...
241223 6:53:40 InnoDB: The InnoDB memory heap is disabled
241223 6:53:40 InnoDB: Mutexes and rw_locks use GCC atomic builtins
241223 6:53:40 InnoDB: Compressed tables use zlib 1.2.7
241223 6:53:40 InnoDB: Using Linux native AIO
241223 6:53:41 InnoDB: Initializing buffer pool, size = 128.0M
241223 6:53:41 InnoDB: Completed initialization of buffer pool
241223 6:53:41 InnoDB: highest supported file format is Barracuda.
241223 6:53:46 InnoDB: Waiting for the background threads to start
241223 6:53:47 Percona XtraDB (Open Source Database Software Support & Services | Percona) 5.5.61-MariaDB-38.13 started; log sequence number 19385320464
241223 6:53:47 [Note] Plugin 'FEEDBACK' is disabled.
241223 6:53:47 [Note] Server socket created on IP: '0.0.0.0'.
241223 6:53:47 [Note] Event Scheduler: Loaded 0 events
241223 6:53:47 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.68-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
241223 6:57:11 [Note] /usr/libexec/mysqld: Normal shutdown
241223 6:57:11 [Note] Event Scheduler: Purging the queue. 0 events
241223 6:57:11 InnoDB: Starting shutdown...
241223 6:57:12 InnoDB: Shutdown completed; log sequence number 19385321042
241223 6:57:12 [Note] /usr/libexec/mysqld: Shutdown complete

241223 06:57:12 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
241223 07:23:07 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
241223 7:23:08 [Note] /usr/libexec/mysqld (mysqld 5.5.68-MariaDB) starting as process 1454 ...
241223 7:23:09 InnoDB: The InnoDB memory heap is disabled
241223 7:23:09 InnoDB: Mutexes and rw_locks use GCC atomic builtins
241223 7:23:09 InnoDB: Compressed tables use zlib 1.2.7
241223 7:23:09 InnoDB: Using Linux native AIO
241223 7:23:09 InnoDB: Initializing buffer pool, size = 128.0M
241223 7:23:09 InnoDB: Completed initialization of buffer pool
241223 7:23:09 InnoDB: highest supported file format is Barracuda.
241223 7:23:13 InnoDB: Waiting for the background threads to start
241223 7:23:14 Percona XtraDB (Open Source Database Software Support & Services | Percona) 5.5.61-MariaDB-38.13 started; log sequence number 19385321042
241223 7:23:15 [Note] Plugin 'FEEDBACK' is disabled.
241223 7:23:15 [Note] Server socket created on IP: '0.0.0.0'.
241223 7:23:15 [Note] Event Scheduler: Loaded 0 events
241223 7:23:15 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.68-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server

Note that I have two versions of this virtual server, one on my real production server and one on a backup server. I am not able to get Plesk up and running on any one of them now :-(

 

[jola]

I'm wondering what is causing the MariaDB to start as mysqld_safe

When I look on log-lines earlier in the log. when my server was working well, it looks as if that was also running in safe mode, here is a part from a week ago:

241215 22:02:34 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
241215 22:02:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
241215 22:02:56 [Note] /usr/libexec/mysqld (mysqld 5.5.68-MariaDB) starting as process 1557 ...
241215 22:02:56 InnoDB: The InnoDB memory heap is disabled
241215 22:02:56 InnoDB: Mutexes and rw_locks use GCC atomic builtins
241215 22:02:56 InnoDB: Compressed tables use zlib 1.2.7
241215 22:02:56 InnoDB: Using Linux native AIO
241215 22:02:56 InnoDB: Initializing buffer pool, size = 128.0M
241215 22:02:56 InnoDB: Completed initialization of buffer pool
241215 22:02:56 InnoDB: highest supported file format is Barracuda.
241215 22:02:56 InnoDB: Waiting for the background threads to start
241215 22:02:57 Percona XtraDB (Open Source Database Software Support & Services | Percona) 5.5.61-MariaDB-38.13 started; log sequence number 19323138210
241215 22:02:57 [Note] Plugin 'FEEDBACK' is disabled.
241215 22:02:57 [Note] Server socket created on IP: '0.0.0.0'.
241215 22:02:57 [Note] Event Scheduler: Loaded 0 events
241215 22:02:57 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.68-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server

 

[WebHostingAce]

Could you please try,

#service mariadb stop

Then,

#service mysql restart

I think it is best to contact Plesk Support team regarding this.

 

[jola]

I contacted Plesk support. I hope they can find the problem. Right now my case is in "AI Triage"... don't know how long it will stay there until a real person looks at it. I hope they do not try to reach me via my normal email address, since that is on one of the sites on the broken server :-/

 

[WebHostingAce]

I do understand your situation.

Plesk Support will help you in this case.

When I look on log-lines earlier in the log. when my server was working well, it looks as if that was also running in safe mode, here is a part from a week ago:

This should have been ok but nothing explain why the MariaDB is not talking the quires in your log file.

Also can you login to MariaDB server with

#plesk db

Then,

in the MariaDB server,

> SHOW DATABASES;