Resolved Innodb corrupt after Plesk update

[Dave W]

Hello,

Plesk Onyx:

• Product: Plesk Onyx 17.8.11 Update #72 , last updated at Nov 6, 2019 03:14 AM
• CentOS Linux release 7.7.1908 (Core)

We've just had a mariadb server fall over due to an update by plesk. We did have settings to update as follows:

• Automatically install Plesk updates (Recommended) Critical security updates are always installed automatically.
• Automatically install updates for third-party components shipped by Plesk
• Automatically install system package updates
  
• Enable safe updates for system packages

I have attached the autoinstaller log and the relevant /var/log/messages

/var/log/messages:

Nov  6 03:14:45 SERVER setroubleshoot: failed to retrieve rpm info for /var/lib/mysql/mysql.sock
Nov  6 03:14:46 SERVER yum[14739]: Updated: MariaDB-client-10.1.42-1.el7.centos.x86_64
Nov  6 03:14:47 SERVER yum[14739]: Updated: galera-25.3.28-1.rhel7.el7.centos.x86_64
Nov  6 03:15:05 SERVER setroubleshoot: failed to retrieve rpm info for /var/lib/mysql/mysql.sock
Nov  6 03:15:07 SERVER yum[14739]: Updated: MariaDB-server-10.1.42-1.el7.centos.x86_64
Nov  6 03:15:08 SERVER yum[14739]: Updated: MariaDB-shared-10.1.42-1.el7.centos.x86_64
Nov  6 03:15:08 SERVER mysqld: 2019-11-06  3:15:08 140505679267584 [Note] /usr/sbin/mysqld: Normal shutdown
Nov  6 03:15:08 SERVER mysqld: 2019-11-06  3:15:08 140505679267584 [Note] Event Scheduler: Purging the queue. 0 events
Nov  6 03:15:10 SERVER mysqld: 2019-11-06  3:15:10 140505040013056 [Note] InnoDB: FTS optimize thread exiting.
Nov  6 03:15:10 SERVER mysqld: 2019-11-06  3:15:10 140505679267584 [Note] InnoDB: Starting shutdown...
Nov  6 03:15:11 SERVER mysqld: 2019-11-06  3:15:11 140505679267584 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
Nov  6 03:15:13 SERVER mysqld: 2019-11-06  3:15:13 140505679267584 [Note] InnoDB: Shutdown completed; log sequence number 16830441939
Nov  6 03:15:14 SERVER mysqld: 2019-11-06  3:15:14 140505679267584 [Note] /usr/sbin/mysqld: Shutdown complete
Nov  6 03:15:14 SERVER mysqld: 2019-11-06  3:15:14 140265811859712 [Note] /usr/sbin/mysqld (mysqld 10.1.42-MariaDB) starting as process 15069 ...
Nov  6 03:15:14 SERVER mysqld: 2019-11-06  3:15:14 140265811859712 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
Nov  6 03:15:14 SERVER mysqld: 2019-11-06  3:15:14 140265811859712 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Nov  6 03:15:14 SERVER mysqld: 2019-11-06  3:15:14 140265811859712 [Note] InnoDB: The InnoDB memory heap is disabled
Nov  6 03:15:14 SERVER mysqld: 2019-11-06  3:15:14 140265811859712 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Nov  6 03:15:14 SERVER mysqld: 2019-11-06  3:15:14 140265811859712 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
Nov  6 03:15:14 SERVER mysqld: 2019-11-06  3:15:14 140265811859712 [Note] InnoDB: Compressed tables use zlib 1.2.7
Nov  6 03:15:14 SERVER mysqld: 2019-11-06  3:15:14 140265811859712 [Note] InnoDB: Using Linux native AIO
Nov  6 03:15:14 SERVER mysqld: 2019-11-06  3:15:14 140265811859712 [Note] InnoDB: Using SSE crc32 instructions
Nov  6 03:15:14 SERVER mysqld: 2019-11-06  3:15:14 140265811859712 [Note] InnoDB: Initializing buffer pool, size = 128.0M
Nov  6 03:15:14 SERVER mysqld: 2019-11-06  3:15:14 140265811859712 [Note] InnoDB: Completed initialization of buffer pool
Nov  6 03:15:14 SERVER mysqld: 2019-11-06  3:15:14 140265811859712 [Note] InnoDB: Highest supported file format is Barracuda.
Nov  6 03:15:15 SERVER mysqld: 2019-11-06 03:15:15 7f922de3d900  InnoDB: Assertion failure in thread 140265811859712 in file dict0dict.cc line 1493
Nov  6 03:15:15 SERVER mysqld: InnoDB: Failing assertion: table->can_be_evicted
Nov  6 03:15:15 SERVER mysqld: InnoDB: We intentionally generate a memory trap.

From /tmp/autoinstaller3.log:

[2019-11-06 03:14:15.817244] Downloading file PHP73_17/php73-cos7-x86_64.inf3: 0%
[2019-11-06 03:14:15.864464] Downloading file PHP73_17/php73-cos7-x86_64.inf3: 100% was finished.
[2019-11-06 03:14:15.864677] ProductInf3FileGuard destroyed. Build didn't change
[2019-11-06 03:14:15.864703] FileFetcher: get file (~empty)/SITEBUILDER_17.8.12/sitebuilder-17.8.12-rhall-all.inf3
[2019-11-06 03:14:15.864743] Downloading file SITEBUILDER_17.8.12/sitebuilder-17.8.12-rhall-all.inf3: 0%
[2019-11-06 03:14:15.934624] Downloading file SITEBUILDER_17.8.12/sitebuilder-17.8.12-rhall-all.inf3: 100% was finished.
[2019-11-06 03:14:15.934818] ProductInf3FileGuard destroyed. Build didn't change
[2019-11-06 04:57:38.968142] Environment locale name is 'C'
[2019-11-06 04:57:39.013439]
[2019-11-06 04:57:39.013474] SourceUrl: original source url is Index of /
[2019-11-06 04:57:39.013783] SourceUrl: fixed url is [URL]http://autoinstall.plesk.com/

Noticed in yum.log that it seems to have tried updating Mariadb twice?

Nov 06 03:15:07 Updated: MariaDB-server-10.1.42-1.el7.centos.x86_64
Nov 06 03:15:08 Updated: MariaDB-shared-10.1.42-1.el7.centos.x86_64

I'm not sure this is a plesk issue, but if anyone can shed some light Id appreciate it.

Rgds
Dave_W

 

[Bitpalast]

@IgorG please take care of this asap

 

[IgorG]

We have related article already - MariaDB does not start after update on November the 5th: Assertion failure in file

 

[Bitpalast]

@IgorG: You have been and still are the best!!! Thank you!!!!!!!!!!!!! I cannot express the amount of gratitude I have here in words.
Applying the article has worked around the issue and let the DB server start again.

I am now highly worried, because on all of our other systems, MariaDB libary updates are pending. How do we know whether they will lead into a desaster of whether they will work?

 

[RobHost]

How to prevent Plesk from doing this upgrade?

We have this in our config, this does not prevent Plesk from updating MariaDB this night on several nodes:

[updates]
systemUpdatesTool = on

 

[RobHost]

Autoinstaller did indeed the upgrade, despite auto updates are off (see attached image). Why is that?

[2019-11-06 03:06:24.080488] Following bootstrapper packages will be installed: (empty)
[2019-11-06 03:06:24.080506]  ----------------
[2019-11-06 03:06:24.080519] Getting packages to installation list: [2019-11-06 03:06:24.080564]   skip package 'ext-letsencrypt-2.8.3-541.noarch' from component letsencrypt - same or newer version of this package is already installed (in system ext-letsencrypt-2.8.3-541.noarch)
[2019-11-06 03:06:24.080612] Following packages will be installed: MariaDB-server-10.1.42-1.el7.centos ext-letsencrypt-2.8.4-547.noarch
 [2019-11-06 03:06:24.080635]  ----------------

 

[mizar]

Hello,

I am now highly worried, because on all of our other systems, MariaDB libary updates are pending.How do we know whether they will lead into a desaster of whether they will work?

Probably it would be some kind of announce from MariaDB side, or just a new version with fix

How to prevent Plesk from doing this upgrade?

Best way is blacklist packages in package manager level by exclude=MariaDB-* in yum.conf

 

[RobHost]

Best way is blacklist packages in package manager level by exclude=MariaDB-* in yum.conf

Can you explain why Plesk actually did the MariaDB upgrade, even when system updates are deactivated? Does Plesk ignore these settings for some mandatory packages?

 

[bulent]

Yep, I had similar problem this morning and restored full proxmox image from 1 day ago and for now disabled plesk updates.

 

[moswak]

MariaDB has removed all affected versions from their repository.

MariaDB has fixed the issue in the following unpublished versions:
10.1.43 [ 23703 ]
10.2.29 [ 23911 ]
10.3.20 [ 23909 ]
10.4.10 [ 23907 ]

I do not know when these will be published

 

[Bitpalast]

In the PUM on one affected machine we are now offered version 10.1.43. Is anyone seeing the same and has anyone tried to update to it? I must admit that I don't dare to at the moment until seeing reports of others or Plesk that the update is now safe and will not break the database again.

 

[moswak]

thanks for the information.

no risk no fun

I have made the update from 10.1.41 to 10.1.43 on a previously affected machine and mariadb has started without errors after the update.

2019-11-08 15:46:58 139944891021568 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.1.43-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server

 

[gennolo]

In the PUM on one affected machine we are now offered version 10.1.43. Is anyone seeing the same and has anyone tried to update to it? I must admit that I don't dare to at the moment until seeing reports of others or Plesk that the update is now safe and will not break the database again.

I just upgraded a couple of dev boxes that required downgrade ... From 10.1.41 to 10.1.43 , no issues so far.

I also had several boxes upgraded to 10.1.42 without any issue.
According to jira issue bug only appeared when databases had foreign keys + full text indexes so not all mariadb server were affected.

 

[themew]

10.1.43 finally made it to the CentOS repos today Index of /10.1/centos7-amd64/rpms/

Should be good to go...

 

[Bitpalast]

"Should be" unfortunately is not good enough for us. On a new server with Obsidian we successfully installed 10.2.29. But the official Plesk support article MariaDB does not start after update on November the 5th: Assertion failure in file has not yet been updated. I'd like to see a "solved, the automatic update to the latest MariaDB version is now safe and functioning" or a similar statement. There must be a reason why the article is not updated yet. Maybe simply for the weekend, but maybe for issues that are still present.

 

[moswak]

I can report that we have updated more than 100 servers so far and there is no problem.

 

[Bitpalast]

Thank you. We have now upgraded our systems, too, and all upgrades worked.

 

[mow]

Can you explain why Plesk actually did the MariaDB upgrade, even when system updates are deactivated? Does Plesk ignore these settings for some mandatory packages?

Because it is a third-party component shipped by Plesk, so the checkbox one line above applies. Which is checked in your screenshot.

 

[Ales]

Because it is a third-party component shipped by Plesk, so the checkbox one line above applies. Which is checked in your screenshot.

But it isn't... or is it? MariaDB is either shipped by the OS or installed/upgraded manually using the packages from upstream repos. In either case, packages do not come from Plesk.

This is either a bug or there is some confusion as to what exactly does the checkbox mean.

@IgorG, could you please clarify?

 

[themew]

Our email looks like MariaDB comes directly from the MariaDB repo

- MariaDB-client 10.1.43-1.el7.centos from mariadb repo (currently installed version: 10.1.41-1.el7.centos from mariadb repo)
- MariaDB-common 10.1.43-1.el7.centos from mariadb repo (currently installed version: 10.1.41-1.el7.centos from mariadb repo)
- MariaDB-server 10.1.43-1.el7.centos from mariadb repo (currently installed version: 10.1.41-1.el7.centos from mariadb repo)
- MariaDB-shared 10.1.43-1.el7.centos from mariadb repo (currently installed version: 10.1.41-1.el7.centos from mariadb repo)
- galera 25.3.28-1.rhel7.el7.centos from mariadb repo (currently installed version: 25.3.26-1.rhel7.el7.centos from mariadb repo)

Could be how they were able to pull 10.1.42.1 so fast so it couldn't be used in a update.

Really glad this was covered by Plesk to avoid confusion.