Issue Under load conditions, PHP-FPM processes increase and php-fpm logs shows "Too many open files"

[Boris]

Server operating system version

Ubuntu 20.04.5 LTS x64

Plesk version and microupdate number

Plesk Obsidian Web Admin Edition Versión 18.0.49 Actualización #2

PROBLEM DESCRIPTION

Under load conditions, PHP-FPM (7.3) processes increase to about 500 at which point it shows to the php-fpm error log "Too many open files" and the web stops serving content until the number of php-fpm processes fpm drops again.
On the server we have PLESK installed with 4 websites (1 website for production and 3 websites for development with the same code an BBDD in each of them).
Server has 96 Gb de RAM and nvme disks.


STEPS TO REPRODUCE

Parameters of configuration file /var/www/vhosts/system/XXXXXXXX/conf/php-fpm.conf is:

[php-fpm-pool-settings]
pm = dynamic
pm.max_children = 1000
pm.start_servers = 100
pm.min_spare_servers = 100
pm.max_spare_servers = 200
pm.max_requests = 2000

Before we have this parameter:

pm.max_children = 500

and got an error like this:
WARNING: [pool XXXXXXXXX] server reached max_children setting (*), consider raising it

and then we go up to 1000 and now we get an error like this:
ERROR: failed to prepare the stderr pipe: Too many open files (24)
Then we add this parameter to php-fpm configuration:
rlimit_files = 65536
and the log kept showing the above error message (Too many open files). Then we raise the value of the parameter again:
rlimit_files = 131072
and now we keep getting the same error in the php-fpm log:
ERROR: failed to prepare the stderr pipe: Too many open files (24)
It seems as if it is ignoring the limit we just imposed on the service php-fpm.

ACTUAL RESULT

Error messages in log /var/www/vhosts/XXXXXX/logs/php-fpm_error_log:

ERROR: failed to prepare the stderr pipe: Too many open files (24)
ERROR: failed to prepare the stderr pipe: Too many open files (24)

When browsing the web at that time, a white screen appears waiting to receive data from the web.

EXPECTED RESULT

That the php-fpm error disappears:

ERROR: failed to prepare the stderr pipe: Too many open files (24)
and that our website is served correctly in thos conditions.

 

[Peter Debik]

This question is about "Too many open files" which is caused by settings in the operating system. You need to at least modify settings in /etc/security/limits.conf for nginx, root, psaadm, mysql, httpd to a high value that fits your requirements. It is also needed to set fs.file-max in /etc/sysctl.conf to a high value. I also recommend to add "LimitNOFILE=<high number>" to the [Service] section of /usr/lib/systemd/system/nginx.service and into /etc/sysconfig/nginx.systemd.

After these changes, the affected services should be restarted. You will also need to reload system daemon with # systemctl --system daemon-reload.

Please at least also check

https://support.plesk.com/hc/en-us/articles/213938485-nginx-fails-to-start-reload-on-a-Plesk-server-Too-many-open-files

https://support.plesk.com/hc/en-us/articles/213938885-MySQL-on-the-server-with-Plesk-shows-the-error-Too-many-open-files

https://support.plesk.com/hc/en-us/articles/213407989-Apache-failed-to-start-24-Too-many-open-files-Init-Can-t-open-server-certificate-file

There may be more to update on the system.

In general: The situation you describe is quite unusual. 500 children of PHP-FPM I think cannot work. Normal values are somewhere between 10 and 30. The maximum number of concurrent Apache processes is 256, while this will certainly lead to a stalled system. Instead, I suggest to rethink your website's software design so that it handles processes with less resources.