• If you are still using CentOS 7.9, it's time to convert to Alma 8 with the free centos2alma tool by Plesk or Plesk Migrator. Please let us know your experiences or concerns in this thread:
    CentOS2Alma discussion

Issue Database server

matbcvo

Basic Pleskian
We have an issue with database server. Some one day all ~100 websites suddenly doesn't work (can't be reached or 504 gateway time out etc). But at same time Plesk works fine, we can log into it. We are able to ping Plesk server. So I tried to restart database server (service mysql restart || service mariadb restart || service mysqld restart) then all websites goes back to up and works nicely. So issue is database server.

We have tried Plesk self-repair tool (Plesk Database and MySQL), it didn't detect any issues.

I checked /etc/my.cnf and added "innodb_buffer_pool_size=8G" (default is 128MB, I though it would help fix this issue) as our dedicated server has 64GB RAM.
Code:
[mysqld]
#bind-address = ::ffff:127.0.0.1
bind-address = ::
local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd

wait_timeout = 31536000
max_allowed_packet=16M

innodb_buffer_pool_size=8G

[mysqldump]
max_allowed_packet=16M

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

I checked /var/log/mariadb/mariadb.log and found something strange ("InnoDB: ERROR: the age of the last checkpoint is 9438524,"...)
Code:
InnoDB: largest such row.
210728 11:55:39  InnoDB: ERROR: the age of the last checkpoint is 9438524,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.
210728 11:56:04  InnoDB: ERROR: the age of the last checkpoint is 9433886,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.
210728 11:56:51  InnoDB: ERROR: the age of the last checkpoint is 9442756,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.
210728 11:57:21  InnoDB: ERROR: the age of the last checkpoint is 9449805,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.
210728 11:57:39  InnoDB: ERROR: the age of the last checkpoint is 9438356,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.
210728 11:57:58  InnoDB: ERROR: the age of the last checkpoint is 9436155,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.
210728 11:58:19  InnoDB: ERROR: the age of the last checkpoint is 9446266,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.
210728 12:16:11 [Note] /usr/libexec/mysqld: Normal shutdown
210728 12:16:11 [Note] Event Scheduler: Purging the queue. 0 events
210728 12:16:11  InnoDB: Starting shutdown...
210728 12:16:12  InnoDB: Waiting for 201 pages to be flushed
210728 12:16:17  InnoDB: Shutdown completed; log sequence number 56775266295252
210728 12:16:17 [Note] /usr/libexec/mysqld: Shutdown complete

210728 12:16:17 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
210728 12:16:18 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
210728 12:16:18 [Note] /usr/libexec/mysqld (mysqld 5.5.68-MariaDB) starting as process 23760 ...
210728 12:16:18 InnoDB: The InnoDB memory heap is disabled
210728 12:16:18 InnoDB: Mutexes and rw_locks use GCC atomic builtins
210728 12:16:18 InnoDB: Compressed tables use zlib 1.2.7
210728 12:16:18 InnoDB: Using Linux native AIO
210728 12:16:18 InnoDB: Initializing buffer pool, size = 8.0G
210728 12:16:18 InnoDB: Completed initialization of buffer pool
210728 12:16:18 InnoDB: highest supported file format is Barracuda.
210728 12:16:19  InnoDB: Waiting for the background threads to start
210728 12:16:20 Percona XtraDB (http://www.percona.com) 5.5.61-MariaDB-38.13 started; log sequence number 56775266295252
210728 12:16:20 [Note] Plugin 'FEEDBACK' is disabled.
210728 12:16:20 [Note] Server socket created on IP: '::'.
210728 12:16:20 [Note] Event Scheduler: Loaded 0 events
210728 12:16:20 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.68-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server

Any suggestions?

CPUIntel(R) Core(TM) i7-7700 CPU @ 3.60GHz (8 core(s))
VersionPlesk Obsidian v18.0.36_build1800210604.22 os_CentOS 7
OSCentOS Linux 7.9.2009 (Core)
 
And, do you use big blobs or texts that exceed 90MB?

You should set
innodb_log_file_size = 1024M
(1/8 of buffer pool size, as mysqltuner would tell you)
 
I was reading about innodb_log_file_size, will change that value. Thank you, will also look into mysqltuner. We do have some sites that read/write large data from/to database server.
 
Unfortunately we are still experiencing same issue, even there is no warnings in mariadb.log.
CPU total usage 20% (not overloaded), RAM total usage 5%, disks write/read currently zero

Code:
[mysqld]
#bind-address = ::ffff:127.0.0.1
bind-address = ::
local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd

wait_timeout = 31536000
max_allowed_packet=16M

innodb_buffer_pool_size=8G
innodb_log_file_size=1G
innodb_buffer_pool_instances=8
innodb_stats_on_metadata=OFF
#table_definition_cache=-1
thread_cache_size=4

[mysqldump]
max_allowed_packet=16M

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

Code:
210729 11:25:38 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
210729 11:25:38 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
210729 11:25:38 [Warning] option 'table_definition_cache': unsigned value 18446744073709551615 adjusted to 524288
210729 11:25:38 [Note] /usr/libexec/mysqld (mysqld 5.5.68-MariaDB) starting as process 30107 ...
210729 11:25:38 InnoDB: The InnoDB memory heap is disabled
210729 11:25:38 InnoDB: Mutexes and rw_locks use GCC atomic builtins
210729 11:25:38 InnoDB: Compressed tables use zlib 1.2.7
210729 11:25:38 InnoDB: Using Linux native AIO
210729 11:25:38 InnoDB: Initializing buffer pool, size = 8.0G
210729 11:25:38 InnoDB: Completed initialization of buffer pool
210729 11:25:38 InnoDB: highest supported file format is Barracuda.
210729 11:25:39  InnoDB: Waiting for the background threads to start
210729 11:25:40 Percona XtraDB (http://www.percona.com) 5.5.61-MariaDB-38.13 started; log sequence number 56909453411196
210729 11:25:40 [Note] Plugin 'FEEDBACK' is disabled.
210729 11:25:40 [Note] Server socket created on IP: '::'.
210729 11:25:40 [Note] Event Scheduler: Loaded 0 events
210729 11:25:40 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.68-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
210729 11:28:57 [Note] /usr/libexec/mysqld: Normal shutdown
210729 11:28:57 [Note] Event Scheduler: Purging the queue. 0 events
210729 11:28:57  InnoDB: Starting shutdown...
210729 11:28:59  InnoDB: Waiting for 201 pages to be flushed
210729 11:29:11  InnoDB: Shutdown completed; log sequence number 56909746804258
210729 11:29:11 [Note] /usr/libexec/mysqld: Shutdown complete

210729 11:29:11 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
210729 11:29:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
210729 11:29:11 [Note] /usr/libexec/mysqld (mysqld 5.5.68-MariaDB) starting as process 24223 ...
210729 11:29:11 InnoDB: The InnoDB memory heap is disabled
210729 11:29:11 InnoDB: Mutexes and rw_locks use GCC atomic builtins
210729 11:29:11 InnoDB: Compressed tables use zlib 1.2.7
210729 11:29:11 InnoDB: Using Linux native AIO
210729 11:29:11 InnoDB: Initializing buffer pool, size = 8.0G
210729 11:29:11 InnoDB: Completed initialization of buffer pool
210729 11:29:11 InnoDB: highest supported file format is Barracuda.
210729 11:29:12  InnoDB: Waiting for the background threads to start
210729 11:29:13 Percona XtraDB (http://www.percona.com) 5.5.61-MariaDB-38.13 started; log sequence number 56909746804258
210729 11:29:13 [Note] Plugin 'FEEDBACK' is disabled.
210729 11:29:13 [Note] Server socket created on IP: '::'.
210729 11:29:13 [Note] Event Scheduler: Loaded 0 events
210729 11:29:13 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.68-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
 
Wait. mysqld 5.5.68-MariaDB / Percona XtraDB 5.5.61-MariaDB-38.13? Isn't that, you know, ancient?
And you're binding to :: (any IP)? I hope you have a firewall so the mysql server isn't exposed to the world because you're bound to have several unpatched CVEs there.
 
Yes, it is ancient version (CentOS 7 is shipped with that), I wonder why Plesk officially does not offer upgrade MariaDB 5.5 to MariaDB 10.X

mysqltuner output contained that:
Code:
-------- CVE Security Recommendations --------------------------------------------------------------
[OK] NO SECURITY CVE FOUND FOR YOUR VERSION
 
Wait. mysqld 5.5.68-MariaDB / Percona XtraDB 5.5.61-MariaDB-38.13? Isn't that, you know, ancient?
And you're binding to :: (any IP)? I hope you have a firewall so the mysql server isn't exposed to the world because you're bound to have several unpatched CVEs there.

The MariaDB package that comes with CentOS 7 still receives updates and security fixes from Redhat until EOL of CentOS 7 (mid 2024). There are no (known) CVEs in the latest 5.5.x MariaDB from CentOS. Version numbers may stay the same as the fixes are backported, see: The Package Versions - Why our package versions are (almost) never bumped up?
 
The MariaDB package that comes with CentOS 7 still receives updates and security fixes from Redhat until EOL of CentOS 7 (mid 2024).
Okay, they backport the security fixes, but there has been a bunch of improvements between 5.5 and 10.x that I doubt were backported too because what's the point of maintaining a separate version when you backport everything?
 
Sure, new features are not backported, only bugfixes and security fixes. So yes, upgrading to a supported version of MariaDB 10.x is definitely a good plan.
 
Back
Top