Resolved MariaDB package updates via Plesk have broken MariaDB and now won't restart

[zigojacko]

Today (6th November 2019), Plesk Obsidian sent out an email notification of new package updates to MariaDB (see below screenshot).

Like usual, I logged in to the Plesk interface to update these thinking nothing of it but the process hung and never completed. I tried to log back into Plesk but I got the 500 server error page (see below) because it was no longer connected to the database.

So I tried to restart MariaDB but it was failing with the following error:

Redirecting to /bin/systemctl start mariadb.service
Job for mariadb.service failed because a fatal signal was delivered to the control process. See "systemctl status mariadb.service" and "journalctl -xe" for details.

The contents of systemctl status mariadb.service is:

● mysql.service - LSB: start and stop MariaDB
   Loaded: loaded (/etc/rc.d/init.d/mysql; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2019-11-06 13:46:55 GMT; 2s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 14969 ExecStart=/etc/rc.d/init.d/mysql start (code=exited, status=1/FAILURE)

Nov 06 13:46:54 hostname.replaced.co.uk systemd[1]: Starting LSB: start and stop MariaDB...
Nov 06 13:46:54 hostname.replaced.co.uk mysql[14969]: Starting MariaDB.191106 13:46:54 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Nov 06 13:46:54 hostname.replaced.co.uk mysql[14969]: 191106 13:46:54 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Nov 06 13:46:55 hostname.replaced.co.uk mysql[14969]: /etc/rc.d/init.d/mysql: line 264: kill: (14991) - No such process
Nov 06 13:46:55 hostname.replaced.co.uk mysql[14969]: ERROR!
Nov 06 13:46:55 hostname.replaced.co.uk systemd[1]: mysql.service: control process exited, code=exited status=1
Nov 06 13:46:55 hostname.replaced.co.uk systemd[1]: Failed to start LSB: start and stop MariaDB.
Nov 06 13:46:55 hostname.replaced.co.uk systemd[1]: Unit mysql.service entered failed state.
Nov 06 13:46:55 hostname.replaced.co.uk systemd[1]: mysql.service failed.

The contents of journalctl -xe is:

Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467422 K  bytes of memory
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Hope that's ok; if not, decrease some variables in the equation.
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Thread pointer: 0x0
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Attempting backtrace. You can use the following information to find out
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: where mysqld died. If you see no messages after this, something went
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: terribly wrong...
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: stack_bottom = 0x0 thread_stack 0x49000
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ceac66c58e]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ceac10954f]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: sigaction.c:0(__restore_rt)[0x7f0d4e8775f0]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: :0(__GI_raise)[0x7f0d4cb48337]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: :0(__GI_abort)[0x7f0d4cb49a28]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(+0x4d3560)[0x55ceabe49560]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(+0xaebf97)[0x55ceac461f97]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(+0xb02603)[0x55ceac478603]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(+0xb02a1d)[0x55ceac478a1d]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(+0xb03976)[0x55ceac479976]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(+0xaf0d50)[0x55ceac466d50]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(+0xb4cd08)[0x55ceac4c2d08]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(+0xb578f2)[0x55ceac4cd8f2]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(+0x984cf3)[0x55ceac2facf3]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(+0xa3b0e8)[0x55ceac3b10e8]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(+0x92a001)[0x55ceac2a0001]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x64)[0x55ceac10bdd4]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(+0x5c59f0)[0x55ceabf3b9f0]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(_Z11plugin_initPiPPci+0x9ba)[0x55ceabf3cb7a]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(+0x4f8bf1)[0x55ceabe6ebf1]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x4ae)[0x55ceabe74dde]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f0d4cb34505]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: /usr/sbin/mysqld(+0x4f280d)[0x55ceabe6880d]
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: information that should help you find out what is causing the crash.
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Writing a core file...
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Working directory at /var/lib/mysql
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Resource Limits:
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Limit                     Soft Limit           Hard Limit           Units
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max cpu time              unlimited            unlimited            seconds
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max file size             unlimited            unlimited            bytes
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max data size             unlimited            unlimited            bytes
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max stack size            8388608              unlimited            bytes
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max core file size        0                    unlimited            bytes
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max resident set          unlimited            unlimited            bytes
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max processes             31088                31088                processes
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max open files            16364                16364                files
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max locked memory         65536                65536                bytes
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max address space         unlimited            unlimited            bytes
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max file locks            unlimited            unlimited            locks
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max pending signals       31088                31088                signals
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max msgqueue size         819200               819200               bytes
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max nice priority         0                    0
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max realtime priority     0                    0
Nov 06 14:07:17 hostname.replaced.co.uk mysqld[15637]: Max realtime timeout      unlimited            unlimited            us
Nov 06 14:07:17 hostname.replaced.co.uk systemd[1]: mariadb.service: main process exited, code=killed, status=6/ABRT
Nov 06 14:07:17 hostname.replaced.co.uk systemd[1]: Failed to start MariaDB 10.3.19 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Nov 06 14:07:17 hostname.replaced.co.uk systemd[1]: Unit mariadb.service entered failed state.
Nov 06 14:07:17 hostname.replaced.co.uk systemd[1]: mariadb.service failed.

Please can anyone advise how to properly debug this and fix so that my database service starts back up again?

 

[mizar]

Hello,

In short - after upgrade your DB server to MariaDB 10.2.28 can refuse to start (if you have manually install the such version) See also MariaDB does not start after update on November the 5th: Assertion failure in file

 

[zigojacko]

Hello,

In short - after upgrade your DB server to MariaDB 10.2.28 can refuse to start (if you have manually install the such version) See also MariaDB does not start after update on November the 5th: Assertion failure in file

Incredible. That's exactly the problem I experienced and I searched the web high and low and went through tons of attempts to get this fixed yesterday and not once did I even come across the page that you've linked to which is exactly what I needed by the looks of it.

Anyhow, in the end, I managed to fix this myself (before seeing your reply above).

In short, after trying tons of things, I think I managed to get somewhere by:

* Backing up entire /var/lib/mysql directory.
* Force stopping any mariadb or mysqld services.
* Completely uninstalling mariadb service.
* Removing the files 'ib_logfile0' and 'ib_logfile1' from the same directory.
* Reinstalling mariadb service.
* Repairing Plesk.

Or at least something along the lines of that... All I know is, it was a nightmare but finally got it sorted.

For anyone else that comes across this issue, definitely follow the resolution that @mizar posted above.