Issue All websites are reported as down by 360 monitoring during night

[IT Ufficio Key-One]

Server operating system version

Almalinux 8.9

Plesk version and microupdate number

18.0.60 #1

Hi, we have a VPS with Almalinux 8.9 where several websites are hosted and monitored via 360 monitoring.

Every night from 03:20 to 04:00 (not exactly but always during this interval), 360 monitor reports that all websites are down (timeout).

I see that during the same interval, the server log rotation occurs. It is something related? On another VPS with the identical configuration (or also some other with still Centos 7.9) , the issue is not happening.

Could you please help?

Thanks.

 

[Kaspar@Plesk]

Are there any other indicators that your websites are off line during that time frame? Error in the domains error log? Errors or warnings in the system logs? Does monitoring show a high resource load during that time frame?

 

[IT Ufficio Key-One]

Are there any other indicators that your websites are off line during that time frame? Error in the domains error log? Errors or warnings in the system logs? Does monitoring show a high resource load during that time frame?

Hi @Kaspar@Plesk , monitoring shows during this period high cpu, disk usage peak, high load avg, memory is almost full.

Mariadb and php-fpm processes have a peak too. Mariadb up to 90%, php-fpm to 70%.

After this period all values return as normal.

In php-fpm logs there are many "database error Lock wait timeout exceeded; try restarting transaction" for all websites
In access_ssl_log of all websites during this period nothing has been traced. There's an "hole" exact during these minutes.

There is no backup active during this period that could interfere.

Thanks.

 

[Kaspar@Plesk]

Hi @Kaspar@Plesk , monitoring shows during this period high cpu, disk usage peak, high load avg, memory is almost full.

Mariadb and php-fpm processes have a peak too. Mariadb up to 90%, php-fpm to 70%.

That's probably why the your websites are offline. Maybe there are scheduled tasks running that trigger processes that are causing high load?

If you need help with investigating what's causing the high load I suggest to contact Plesk support for help. To sign-in to support please go to https://support.plesk.com

If you bought your license from a reseller, your reseller should provide support for you. If the reseller does not provide support, here is an alternative:
https://support.plesk.com/hc/en-us/articles/12388090147095-How-to-get-support-directly-from-Plesk

 

[IT Ufficio Key-One]

Thanks for your reply. After checking again the time when this happens, I'm supposing that this issue could be related to log rotation.

All websites are reported as down exactly starting at 03:34 and they're back just after 03:38 (360 Monitoring takes a while more to check them again and tell they're back),as I can see from their logs.

At 03:38 all logs are rotated and are new, there are no more warnings or errors on newly created i.e. plesk-phpxx-fpm/error.log and everything seems to be back normal.

Do you think this could be possible?

Here below an excerpt of /var/log/cron

May 31 03:34:05 k1nvps02 run-parts[716509]: (/etc/cron.daily) starting logrotate
May 31 03:36:01 k1nvps02 CROND[718254]: (psaadm) CMD (/usr/local/psa/admin/bin/php -dauto_prepend_file=sdk.php '/usr/local/psa/admin/plib/modules/revisium-antivirus/scripts/ra_executor_run.php')
May 31 03:38:01 k1nvps02 CROND[718808]: (psaadm) CMD (/usr/local/psa/admin/bin/php -dauto_prepend_file=sdk.php '/usr/local/psa/admin/plib/modules/wp-toolkit/scripts/maintenance.php')
May 31 03:38:01 k1nvps02 CROND[718807]: (psaadm) CMD (/usr/local/psa/admin/bin/php -dauto_prepend_file=sdk.php '/usr/local/psa/admin/plib/modules/sslit/scripts/panel-notifications.php')
May 31 03:38:02 k1nvps02 crontab[718835]: (root) LIST (psaadm)
May 31 03:38:25 k1nvps02 run-parts[719541]: (/etc/cron.daily) finished logrotate

 

[Kaspar@Plesk]

I suppose that's possible, but only if you would enormous log file. If you search for large files on the server with command below, are there any log files returned? (It might take a while before the command finishes)

find / -type f -size +200M -exec du -h {} + 2>/dev/null | sort -r -h