• If you are still using CentOS 7.9, it's time to convert to Alma 8 with the free centos2alma tool by Plesk or Plesk Migrator. Please let us know your experiences or concerns in this thread:
    CentOS2Alma discussion

Question Logrotate error: file size changed while zipping

Pleskie

Regular Pleskian
Last night I received this email from my freshly installed server:

Subject: Cron <root@myserver> [ ! -f /etc/cron.hourly/0anacron ] && run-parts /etc/cron.daily

Body:

/etc/cron.daily/logrotate:

error: Compressing program wrote following message to stderr when compressing log /var/log/mariadb/mariadb.log-20210521:
gzip: stdin: file size changed while zipping

What does cause this error? Should I take any kind of action now?
 
This essentially means that your MariaDB logfile was rotated (renamed to mariadb.log-20210521) without instructing MariaDB to flush the logs (close the log file and start a new one), so MariaDB was still having that file opened for writing when it was being gzipped.

When you rename a file on Linux the file descriptor/inode does not change. So any process that had a file opened for writing will continue to write to the file even when it now has a new name.

Solution: You need to check your MariaDB logrotation settings (probably in /etc/logrotate.d/mariadb or /etc/logrotate.d/mysql) and make sure you flush the logs right after rotation.
You can find an example here: Rotating Logs on Unix and Linux
 
Hello @Monty, thanks for your fast and clear reply.

There is no mysql file but there is /etc/logrotate.d/mariadb

These are the contents of this file:

Code:
# This logname can be set in /etc/my.cnf
# by setting the variable "log-error"
# in the [mysqld] section as follows:
#
# [mysqld]
# log-error=/var/log/mariadb/mariadb.log

/var/log/mariadb/mariadb.log {
        create 600 mysql mysql
        notifempty
        daily
        rotate 3
        missingok
        compress
    postrotate
    # just if mysqld is really running
        if [ -e /run/mariadb/mariadb.pid ]
        then
           kill -1 $(</run/mariadb/mariadb.pid)
        fi
    endscript
}

Can you tell me how I should adjust this code to solve the issue? What changes should I make? And should I restart a service afterwards or will it work immediately after changing the code?

This is all new to me so hopefully you can help me out. Thanks in advance!!
 
Try to replace the section between "postrotate" and "endscript" with the example from the link that I've posted above. So something like this:
Code:
postrotate
        # just if mysqld is really running
        if test -x /usr/bin/mysqladmin && \
           /usr/bin/mysqladmin ping &>/dev/null
        then
           /usr/bin/mysqladmin --local flush-error-log \
              flush-engine-log flush-general-log flush-slow-log
        fi
endscript

No need to restart anything. Logrotate will process the file at the next run of the logrotate cronjob (usually in /etc/cron.daily/logrotate).
So just wait and see if this fixes your issue.
 
Thanks again @Monty

I changed the original code with your replacement code.

How do I know if it worked? Just wait 1 day and see if there will be any errors?
 
Yes, wait. If you see that your log file was rotated properly and you did not get any error mails then everything is most likely OK.
If you still get errors or other errors, please post your full OS and Plesk version
 
Thanks @Monty, I very much appreciate your help!

I'll keep an eye on it. I will check if my log rotates properly without any errors. If not, I'll get back to you. Thanks for now! :)
 
Hi @Monty

Well I'm back sooner than expected :)

I received a new mail with another error. Apparently the changed code doesn't work correctly. This is the error message in the email:

Code:
/etc/cron.daily/logrotate:

  /usr/bin/mysqladmin: connect to server at 'localhost' failed
 error: 'Access denied for user 'root'@'localhost' (using password: NO)'
 error: error running non-shared postrotate script for /var/log/mariadb/mariadb.log of '/var/log/mariadb/mariadb.log '

Any idea what to do now?
 
Hello @Monty ,

I checked the intructions, but I don't have a folder /etc/mysql on my system so I can't execute these instructions.

Besides that, it seems to be a resolution to a problem that should have been fixed in 2018 and 2019. The problem I am having is with a brand new Plesk Obsidian install. So I think this solution is not relevant.

I was searching the internet and found this:



It seems to be a known bug and it seems to have been solved with an update. However I don't know how to apply the update. My version of MariaDB is 10.3.28. When I try to update by using "dnf update mariadb" I get a message saying there is nothing to do. There is no update available.

If I understand correctly the second link says there is a version 10.5 available. Any idea how I can apply that update?
 
So what is your OS? If you don't have /etc/mysql then please check the contents of /etc/my.cnf.d (default for Redhat based systems)

You can upgrade MariaDB according to the instructions here:  How to upgrade MySQL 5.5 to 5.6/5.7 or MariaDB 5.5 to 10.x on Linux?

But I suggest you should only do this as a last resort, as such an upgrade is not an easy task. I think it's better when you simply disable logrotation for the MariaDB logs for now (by removing /etc/logrotate.d/mysql or commenting the content of that file). MariaDB logs usually don't grow very rapidly (mine is less than 200mb after 3.5 years of operation).
 
Hi @Monty

>> So what is your OS? If you don't have /etc/mysql then please check the contents of /etc/my.cnf.d (default for Redhat based systems)

My OS is AlmaLinux. First I installed CentOS 8. Then immediately switched to AlmaLinux. It's a clean install and not in use yet, so I could test to update to MariaDB 10.5.

In the directory you mentioned I see a file "mariadb-server.conf".

I still have a few questions:

Question 1

I've read somewhere that in the original code (see my post #3) I could use 'delaycompress' right after the 'compress' command as a possible solution. So the code would be like this:

Apache config:
/var/log/mariadb/mariadb.log {
        create 600 mysql mysql
        notifempty
        daily
        rotate 3
        missingok
        compress
        delaycompress
    postrotate
    # just if mysqld is really running
        if [ -e /run/mariadb/mariadb.pid ]
        then
           kill -1 $(</run/mariadb/mariadb.pid)
        fi
    endscript
}

Would that be a good solution?

Question 2

The link you gave me in your previous post states: "By default, CentOS 8 is shipped with MariaDB 10.3."

Does this mean that if I would not update MariaDB manually, the version will always be 10.3? Or will Plesk or CentOS 8 (or in my case AlmaLinux) one day automatically update MariaDB to version 10.5?

Question 3

The instructions in the link you gave me in your previous post tell to create a new MariaDB.repo file. One of the lines you need to insert into this file is:

Code:
baseurl = http://yum.mariadb.org/10.5/centos8-amd64

Does that mean when I'm updating my files in the future, it will only look for upgrades for version 10.5? And does it mean it won't update to version 10.6 or version 11 automatically (whenever available)? (And if so is that a good thing or not?)
 
Please be aware: Plesk does NOT (yet) support conversion from CentOS 8 to Almalinux. Only new installations of Almalinux are supported for now:

Question 1: "delaycompress" simply means that the rotated logfile will only be compressed after the next rotation. But this does not solve your issue, as the rotated logfile will remain in use by MariaDB until you flush the logs or restart the service.

Question 2: Your version of MariaDB will remain at 10.3 for the whole lifecycle of your CentOS/Almalinux version. When the upstream vendor RedHat releases an OS it freezes the versions of the included packages within that OS version. See also: The Package Versions - Why our package versions are (almost) never bumped up?

So if you want to use a newer version of MariaDB than the one that comes with CentOS/Almalinux then you need to install a third-party repo (like yum.mariadb.org) and use this to install/upgrade your package

Question 3: If you use the repo URL in your example then your MariaDB packages will always remain at version 10.5. To upgrade to a newer version you'd have to change the repo URL to 10.6 (instead of 10.5). Please be advised that you should always check the Plesk software compatibility list before doing so:
 
Hello @Monty

>> Please be aware: Plesk does NOT (yet) support conversion from CentOS 8 to Almalinux. Only new installations of Almalinux are supported for now:

Yes, I'm aware of it. :) That's why I did a new install of CentOS 8, switched to AlmaLinux and then installed Plesk.

Question 1

Okay. So then the best way to solve the problem seems to be an update to MariaDB version 10.5 then. Right? In this version the problem should be solved.

Question 2

Ah okay, I didn't know that. Quite weird actually. So IF CentOS 8 would have been supported until 2029 (which it is not, I know) it would still use MariaDB 10.3 in 2029?

Question 3

Okay I get it. But if I understand you correctly, if I update to version 10.5 then (under normal circumstances) I would 'never' have to update to a newer version?
 
Hi @Monty

I hope you can give me some advice.

First question. A couple of days ago I upgraded to MariaDB 10.5. I verified and everything seems to work fine.

However I noticed the mariadb log has not started rotating. Do I need to do something to restart the logrotation?

Second question. In my messages log I sometimes see warnings like this:

[Warning] Aborted connection 19635 to db: 'psa' user: 'admin' host: 'localhost' (Got timeout reading communication packets)

Do you know what causes these warnings?
 
log rotation is only done when the all the "triggers" in /etc/logrotate.d/mariadb are satisfied. For example, if the log file is empty then it won't be rotated when you have "notifempty" in your logrotate config.

You can always run logrotate in debug mode to see what would be done (but without actually doing anything) by running logrotate -d /etc/logrotate.d/mariadb (or whatever the path to your mariadb logrotate config is)

Regarding the aborted connection errors: That can have many reasons. Most likely it's due to metadata locks due to unoptimized client databases. If you only get that error occasionally then I wouldn't worry too much about it.
Otherwise check those:
 
Thanks. I'm going to do a reinstall now and install the new MariaDB before isnstalling Plesk. Think that will do better ...
 
Hello @Monty,

Could you please help me one more time?

I reinstalled my server. This time AFTER installing the OS and BEFORE installing Plesk, I installed MariaDB version 10.5.

When I SSH this command:

mysqld --version

I get this result:

mysqld Ver 10.5.10-MariaDB for Linux on x86_64 (MariaDB Server)

So it seems everything is going allright and MariaDB is working correctly. So far so good I would think.

Here is my question:

In my first post of this thread I described a problem with compressing logs in directory /var/log/mariadb/.

The strange thing is ... after reinstalling (the way I described above) the directory /var/log/mariadb/ no longer exists.

Please check post number #3 in this thread. In my first install the file /etc/logrotate.d/mysql did NOT exist, but /etc/logrotate.d/mariadb did exist.

However, after the reinstall (as described above) /etc/logrotate.d/mariadb does NOT exist, but /etc/logrotate.d/mysql does exist! So now it's exactly the other way round!

The contents of the file /etc/logrotate.d/mysql are:

Code:
# This logname can be set in /etc/my.cnf
# by setting the variable "log-error"
# in the [mysqld] section as follows:
#
# [mysqld]
# log-error=/var/lib/mysql/mysqld.log
#
# If the root user has a password you have to create a
# /root/.my.cnf configuration file with the following
# content:
#
# [mysqladmin]
# password = <secret>
# user= root
#
# where "<secret>" is the password.
#
# ATTENTION: This /root/.my.cnf should be readable ONLY
# for root !

/var/lib/mysql/mysqld.log {
        # create 600 mysql mysql
        notifempty
        daily
        rotate 3
        missingok
        compress
    postrotate
    # just if mariadbd is really running
    if test -x /usr/bin/mysqladmin && \
       /usr/bin/mysqladmin ping &>/dev/null
    then
       /usr/bin/mysqladmin --local flush-error-log \
              flush-engine-log flush-general-log flush-slow-log
    fi
    endscript
}

When I use the code logrotate -d /etc/logrotate.d/mysql you gave me earlier, I get this result:

Code:
Reading state from file: /var/lib/logrotate/logrotate.status
Allocating hash table for state file, size 64 entries
Creating new state (this line is repeated about 40 times)

Handling 1 logs

rotating pattern: /var/lib/mysql/mysqld.log  after 1 days (3 rotations)
empty log files are not rotated, old logs are removed
considering log /var/lib/mysql/mysqld.log
  log /var/lib/mysql/mysqld.log does not exist -- skipping

In the directory /var/log/ there are no log files for mysql or mariadb although the server has been running for a couple of days now.

I hope you can still follow me :)

In short:

- First install: Old MariaDB sometimes gave errors when rotating. It rotated every night.

- Reinstall: I installed MariaDB 10.5. It seems to be working. However logrotate does not seem to be configured anymore. There are no logs.

What should I be doing? I don't know what to do.

By the way ... when I enter the SSH command plesk db I get this result:

Code:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is xxxxx
Server version: 10.5.10-MariaDB MariaDB Server

So it seems to me Plesk is successfully using MariaDB.

I don't get it ... first there is MariaDB logrotate ... now it seems gone, although MariaDB seems to be functioning normally ... :(:confused:
 
@Pleskie FWIW Amongst all of the other server changes that you've made, you've updated MariaDB from a previous release to 10.5.10.
Along the way, things were changed by MariaDB. Here's an extract from the MariaDB Notification: mysql_install_db

MariaDB starting with 10.4.6

From MariaDB 10.4.6, mariadb-install-db is a symlink to mysql_install_db.

MariaDB starting with 10.5.2

From MariaDB 10.5.2, mysql_install_db is the symlink, and mariadb-install-db the binary name.
Here's an extract from the Percona reference page: Exciting and New Features in MariaDB 10.5 - Percona Database Performance Blog

Binaries Name Changed to mariadb​

All binaries are now changed to “mariadb” from “mysql”, with symlinks for the corresponding mysql command.

Example:
  • “mysql” is now “mariadb”
  • “mysqldump” is now “mariadb-dump”
  • “mysqld” is now “mariadbd”
  • “mysqld_safe” is now “mariadbd-safe”
That's the main reason for all of the name changes that you've seen (we went through this process as we moved through MariaDB 10.3.* > 10.4.* > 10.5.*)

In a previous logrotation thread, we'd mentioned that not all logs & log rotations are directly controlled by Plesk (well, not by default anyway...) and that you can apply your own customisations to achieve what you specifically want, but, that you'll need to test, then alter & re-test (on some occasions) in order to get there.

Again just FWIW using MariaDB 10.5.10, as you are doing now, we curently use:
/etc/logrotate.d/mysql-server to specify the logrotation details for all of the logs that we've chosen for MariaDB (MySQL) logs
All of those ^^ logs are provided and rotated in here: /var/log/mysql
The previous /etc/mysql/my.cnf is now just a symlink to /etc/mysql/mariadb.cnf

There's a large amount of customization that you can apply to MariaDB (MySQL). All of this is covered online by MariaDB & for logrotation etc you can start HERE for reference. That's the same link / starting point for MariaDB logrotation that @Monty posted earlier in this thread and everything you need is covered or linked from there. Can't comment on any subtle differences that you'll experience when using AlmaLinux as we've been on Ubuntu OS for some time now.
 
Back
Top