Resolved 18.0.40 - killed DB as well... all sites now down

[Matt N]

so further to my issues with Postfix/Qmail etc...

I've rebooted the server and now, plesk does not work, mariadb is not working... not able to get anything up.... All customer sites are down.

when starting the Mariadb service, it stops with the following :

Starting mariadb (via systemctl): Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
[FAILED]

when attempting to start the service:

● mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf, tokudb.conf
Active: failed (Result: exit-code) since Mon 2021-11-29 20:59:39 AEDT; 386ms ago
Process: 26687 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 26658 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 26655 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Main PID: 26687 (code=exited, status=1/FAILURE)

Nov 29 20:59:38 server.domain.com.au systemd[1]: Starting MariaDB database server...
Nov 29 20:59:39 server.domain.com.au systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
Nov 29 20:59:39 server.domain.com.au systemd[1]: Failed to start MariaDB database server.
Nov 29 20:59:39 server.domain.com.au systemd[1]: Unit mariadb.service entered failed state.
Nov 29 20:59:39 server.domain.com.au systemd[1]: mariadb.service failed.

results from journalctl command:

Nov 29 21:01:33 server.domain.com.au systemd[1]: Stopped Plesk Web Socket Service.
-- Subject: Unit plesk-web-socket.service has finished shutting down
-- Defined-By: systemd
-- Support: systemd-devel Info Page
--
-- Unit plesk-web-socket.service has finished shutting down.
Nov 29 21:01:33 server.domain.com.au systemd[1]: Starting MariaDB database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: systemd-devel Info Page
--
-- Unit mariadb.service has begun starting up.
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [Note] libgovernor.so found
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [Note] All governors functions found too
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [ERROR] Governor not connected
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [Note] All governors lve functions found too
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [Note] /usr/sbin/mysqld (mysqld 10.2.40-MariaDB-cll-lve) starting as process 29292 ...
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [Note] InnoDB: Uses event mutexes
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [Note] InnoDB: Compressed tables use zlib 1.2.7
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [Note] InnoDB: Using Linux native AIO
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [Note] InnoDB: Number of pools: 1
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [Note] InnoDB: Using SSE2 crc32 instructions
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [Note] InnoDB: Completed initialization of buffer pool
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739235571456 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [Note] InnoDB: Highest supported file format is Barracuda.
Nov 29 21:01:33 server.domain.com.au mysqld[29292]: 2021-11-29 21:01:33 139739947456768 [Note] InnoDB: Starting crash recovery from checkpoint LSN=451518713716

 

[TomBoB]

have you tried to run a complete repair ?

# plesk installer --select-release-current --reinstall-patch --upgrade-installed-components

 

[Matt N]

yep... and it just informs me


<snip>
Downloading file PHP55_17/php55-cos7-x86_64.plesk17.inf3: 0%
Downloading file PHP55_17/php55-cos7-x86_64.plesk17.inf3: 100% was finished.
Downloading file PHP56_17/php56-cos7-x86_64.plesk17.inf3: 0%
Downloading file PHP56_17/php56-cos7-x86_64.plesk17.inf3: 100% was finished.
Downloading file PHP70_17/php70-cos7-x86_64.plesk17.inf3: 0%
Downloading file PHP70_17/php70-cos7-x86_64.plesk17.inf3: 100% was finished.
Downloading file pool/PHP_7.1.33_98/php71-cos7-x86_64.inf3: 0%
Downloading file pool/PHP_7.1.33_98/php71-cos7-x86_64.inf3: 100% was finished.
Downloading file pool/PHP_7.2.34_151/php72-cos7-x86_64.inf3: 0%
Downloading file pool/PHP_7.2.34_151/php72-cos7-x86_64.inf3: 100% was finished.
Downloading file PHP73_17/php73-cos7-x86_64.inf3: 0%
Downloading file PHP73_17/php73-cos7-x86_64.inf3: 100% was finished.
Downloading file PHP74_17/php74-cos7-x86_64.inf3: 0%
Downloading file PHP74_17/php74-cos7-x86_64.inf3: 100% was finished.
Downloading file PHP80_17/php80-cos7-x86_64.inf3: 0%
Downloading file PHP80_17/php80-cos7-x86_64.inf3: 100% was finished.
Downloading file SITEBUILDER_18.0.29/sitebuilder-18.0.29-rhall-all.inf3: 0%
Downloading file SITEBUILDER_18.0.29/sitebuilder-18.0.29-rhall-all.inf3: 100% was finished.
gpg-pubkey-914bdf7e-55c05220 gpg(Plesk Team <[email protected]>)
gpg-pubkey-5ebd2744-418ffac9 gpg(Atomic Rocket Turtle <[email protected]>)
gpg-pubkey-8608cb71-4acc7220 gpg(CloudLinux Packager (CloudLinux Key) <[email protected]>)
gpg-pubkey-f4a80eb5-53a7ff4b gpg(CentOS-7 Key (CentOS 7 Official Signing Key) <[email protected]>)
gpg-pubkey-7fac5991-4615767f gpg(Google, Inc. Linux Package Signing Key <[email protected]>)
gpg-pubkey-4520afa9-50ab914c gpg(Atomicorp (Atomicorp Official Signing Key) <[email protected]>)
gpg-pubkey-352c64e5-52ae6884 gpg(Fedora EPEL (7) <[email protected]>)
gpg-pubkey-d38b4796-570c8cd3 gpg(Google Inc. (Linux Packages Signing Authority) <[email protected]>)
gpg-pubkey-24098cb6-58875d52 gpg(Grafana <[email protected]>)
DB query failed: SQLSTATE[HY000] [2002] No such file or directory

Loaded plugins: fastestmirror, protectbase, rhnplugin
This system is receiving updates from CLN.
Detecting installed product components.
gpg-pubkey-5ebd2744-418ffac9 gpg(Atomic Rocket Turtle <[email protected]>)
gpg-pubkey-8608cb71-4acc7220 gpg(CloudLinux Packager (CloudLinux Key) <[email protected]>)
gpg-pubkey-f4a80eb5-53a7ff4b gpg(CentOS-7 Key (CentOS 7 Official Signing Key) <[email protected]>)
gpg-pubkey-7fac5991-4615767f gpg(Google, Inc. Linux Package Signing Key <[email protected]>)
gpg-pubkey-4520afa9-50ab914c gpg(Atomicorp (Atomicorp Official Signing Key) <[email protected]>)
gpg-pubkey-352c64e5-52ae6884 gpg(Fedora EPEL (7) <[email protected]>)
gpg-pubkey-d38b4796-570c8cd3 gpg(Google Inc. (Linux Packages Signing Authority) <[email protected]>)
gpg-pubkey-24098cb6-58875d52 gpg(Grafana <[email protected]>)
gpg-pubkey-914bdf7e-55c05220 gpg(Plesk Team <[email protected]>)
Retrieving information about the installed license key...
Loaded plugins: fastestmirror, protectbase, rhnplugin
This system is receiving updates from CLN.
You already have the latest version of product(s) and all the selected components installed. Installation will not continue.

 

[TomBoB]

might sound odd but
"DB query failed: SQLSTATE[HY000] [2002] No such file or directory"
happens when you're out of server space

can you confirm you got space left?

 

[Matt N]

output of df -H

Filesystem Size Used Avail Use% Mounted on
devtmpfs 17G 0 17G 0% /dev
tmpfs 17G 0 17G 0% /dev/shm
tmpfs 17G 77M 17G 1% /run
tmpfs 17G 0 17G 0% /sys/fs/cgroup
/dev/mapper/vg-root 976G 128G 799G 14% /
none 17G 4.1k 17G 1% /var/lve/dbgovernor-shm
/dev/md0 508M 208M 274M 44% /boot
tmpfs 3.4G 0 3.4G 0% /run/user/0

doesn't look like a disk space issue

 

[Bitpalast]

What are the startup messages of MariaDB in /var/log/messages?

 

[Matt N]

@Peter Debik

/var/log/messages shows the following detail for Mariadb:

Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] libgovernor.so found
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] All governors functions found too
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [ERROR] Governor not connected
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] All governors lve functions found too
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] /usr/sbin/mysqld (mysqld 10.2.40-MariaDB-cll-lve) starting as process 873538 ...
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: Uses event mutexes
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: Compressed tables use zlib 1.2.7
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: Using Linux native AIO
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: Number of pools: 1
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: Using SSE2 crc32 instructions
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: Completed initialization of buffer pool
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139749310289664 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: Highest supported file format is Barracuda.
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: Starting crash recovery from checkpoint LSN=451518713716
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: 128 out of 128 rollback segments are active.
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: Creating shared tablespace for temporary tables
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: Waiting for purge to start
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] InnoDB: 5.7.35 started; log sequence number 451518714942
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139749163529984 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] Plugin 'FEEDBACK' is disabled.
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] Recovering after a crash using tc.log
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] Starting crash recovery...
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] Crash recovery finished.
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139749163529984 [Note] InnoDB: Buffer pool(s) load completed at 211130 7:17:45
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [Note] Server socket created on IP: '127.0.0.1'.
Nov 30 07:17:45 marvin mysqld: 2021-11-30 7:17:45 139750015498496 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist
Nov 30 07:17:45 marvin systemd: mariadb.service: main process exited, code=exited, status=1/FAILURE
Nov 30 07:17:45 marvin systemd: Failed to start MariaDB database server.
Nov 30 07:17:45 marvin systemd: Unit mariadb.service entered failed state.
Nov 30 07:17:45 marvin systemd: mariadb.service failed.
Nov 30 07:17:45 marvin systemd: Started Plesk Web Socket Service.

I am assuming that because the CloudLinux MySQL Governor didn't load, that that is the reason for MySQL/Mariadb not starting.

 

[Matt N]

found this article while waiting for something to happen. Gave it a go. and at least MySQL/MariaDB is now up and running again and services that were relying on it (websites) are now working again..

https://support.plesk.com/hc/en-us/articles/360018522460-MySQL-service-fails-to-start-on-a-Plesk-server-Table-mysql-user-doesn-t-exist