Issue nginx 24: Too many open files

[sebgonzes]

Hello

In some server that have 700 domains, I have the error with nginx :


nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: [emerg] open() "/var/www/vhosts/system/xxxxxxxxx/logs/proxy_access_ssl_log" failed (24: Too many open files)
nginx: configuration file /etc/nginx/nginx.conf test failed

I try this support article : Nginx fails: Too many open files also with 8192, and same error.

I increase fs.file-max = 70000 in sysctl.conf, I increase /etc/security/limits.conf limits


root soft nofile 64000
root hard nofile 64000
nginx soft nofile 64000
nginx hard nofile 64000

also in /etc/security/limits.d/20-nproc.conf
* soft nproc 64000
root soft nproc unlimited

Also change the daemon :

cat /etc/systemd/system/nginx.service.d/local.conf
[Service]
LimitNOFILE=128104

See these limits

# ps -axwwww | grep nginx
17564 ? Ss 0:00 nginx: master process /usr/sbin/nginx
17565 ? S 0:00 nginx: worker process
17567 ? S 0:00 nginx: worker process
17569 ? S 0:00 nginx: worker process
17570 ? S 0:00 nginx: worker process
22232 pts/0 S+ 0:00 grep --color=auto nginx

# cat /proc/17564/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 128104 128104 processes
Max open files 128104 128104 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 128104 128104 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us


Error still present... I don't know what more I can change...

 

[Bitpalast]

Create a /etc/nginx/ulimit.global_params file and enter
worker_rlimit_nofile 64000;
into it. If the worker_rlimit_nofile entry is present in /etc/nginx/nginx.conf omit this step.

Increase the general maximum file descriptor value:
# vi /etc/sysctl.conf
Add/modify:
fs.file-max = 64000

Edit /etc/security/limits.conf and add:
nginx soft nofile 64000
nginx hard nofile 64000

echo 'NGINX_ULIMIT="-n 64000"' >> /etc/sysconfig/nginx

Edit /usr/lib/systemd/system/nginx.service and add a line in the [Service] section:
LimitNOFILE=64000

In /etc/sysconfig/nginx.systemd add:
LimitNOFILE=64000

Add the line ulimit -n 64000 at the beginning of the /usr/local/psa/admin/sbin/nginx-config script:
#!/usr/bin/env bash
ulimit -n 64000

Reload system daemon:
# systemctl --system daemon-reload
# sysctl -p

Restart sw-cp-server and nginx:
# /etc/init.d/sw-cp-server restart
# /etc/init.d/nginx restart

 

[sebgonzes]

Create a /etc/nginx/ulimit.global_params file and enter
worker_rlimit_nofile 64000;
into it. If the worker_rlimit_nofile entry is present in /etc/nginx/nginx.conf omit this step.

Increase the general maximum file descriptor value:
# vi /etc/sysctl.conf
Add/modify:
fs.file-max = 64000

Edit /etc/security/limits.conf and add:
nginx soft nofile 64000
nginx hard nofile 64000

echo 'NGINX_ULIMIT="-n 64000"' >> /etc/sysconfig/nginx

Edit /usr/lib/systemd/system/nginx.service and add a line in the [Service] section:
LimitNOFILE=64000

In /etc/sysconfig/nginx.systemd add:
LimitNOFILE=64000

Add the line ulimit -n 64000 at the beginning of the /usr/local/psa/admin/sbin/nginx-config script:
#!/usr/bin/env bash
ulimit -n 64000

Reload system daemon:
# systemctl --system daemon-reload
# sysctl -p

Restart sw-cp-server and nginx:
# /etc/init.d/sw-cp-server restart
# /etc/init.d/nginx restart

Hello

I have do it, and still have same problem... nginx start fine, but test failed...


# service sw-cp-server restart
Restarting sw-cp-server (via systemctl): [ OK ]
# service nginx restart
Restarting nginx (via systemctl): [ OK ]
# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: [emerg] open() "/var/www/vhosts/system/xxxxxxxxx.com/logs/proxy_access_ssl_log" failed (24: Too many open files)
nginx: configuration file /etc/nginx/nginx.conf test failed

 

[sebgonzes]

Hello

I have do it, and still have same problem... nginx start fine, but test failed...


# service sw-cp-server restart
Restarting sw-cp-server (via systemctl): [ OK ]
# service nginx restart
Restarting nginx (via systemctl): [ OK ]
# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: [emerg] open() "/var/www/vhosts/system/xxxxxxxxx.com/logs/proxy_access_ssl_log" failed (24: Too many open files)
nginx: configuration file /etc/nginx/nginx.conf test failed

I am in CloudLinux 7, but understand that there is no relation with this...

 

[Bitpalast]

Could it be possible that the overall number of open files exceeds your allotment? Test it with
# lsof | wc -l

 

[sebgonzes]

Could it be possible that the overall number of open files exceeds your allotment? Test it with
# lsof | wc -l

lsof | wc -l
170850

Thanks for your help...

 

[Bitpalast]

According to your post above
Max open files 128104 128104 files
you have many more open files than what is allowed on your server.

 

[sebgonzes]

According to your post above
Max open files 128104 128104 files
you have many more open files than what is allowed on your server.

Understand that problem are the "Max processes", I am able to increase Max open files but don't find where do it with Max processes...

 

[sebgonzes]

cat /proc/2858/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 128104 128104 processes
Max open files 256104 256104 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 128104 128104 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: [emerg] open() "/var/www/vhosts/system/xxxxxxxxx.com/logs/proxy_access_ssl_log" failed (24: Too many open files)
nginx: configuration file /etc/nginx/nginx.conf test failed

 

[MarkM]

Since Plesk 12.5, there's a tool the should be raising the open files limits automatically.... can you check if selinux is enabled?

How do I disable SELinux?

 

[sebgonzes]

Since Plesk 12.5, there's a tool the should be raising the open files limits automatically.... can you check if selinux is enabled?

How do I disable SELinux?

Selinux is disable... is always the first that I do :


]# sestatus
SELinux status: disabled

 

[sebgonzes]

What I can't explain, is why the nginx -t fail, and why nginx can restart without any problem...

 

[sebgonzes]

The solucion in our case...

# ulimit -Hn
4096
# ulimit -Sn 200000

# ulimit -Sn
1024
# ulimit -Sn 200000
# ulimit -Sn
200000
# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

How fix it defintively to not reset this in an reboot?

 

[JonasF]

Sometimes all these solutions dont work. I had this problem on CentOS 8 and it turned out SELinux was preventing the

changed limits. The following command helped: setsebool -P httpd_setrlimit 1

 

[Ayki]

According to your post above
Max open files 128104 128104 files
you have many more open files than what is allowed on your server.

Hi
I have run number of Jmeter tests on my plesk server recently to stress/load test the configuration. The Nginx 24 - too many files open error shuts down the web server. During the test there are long period no more new logins to site can be made with nginx error 502.
Shall I start a new discussion and document the issue ? Or would you be able to guide me to trouble shoot the issue?
I think the root cause is the Linux ulimit for open files. Possiblly to increase the soft limit for plesk admin to number higher than 1024.
However httpd seams to have multiple instances running with over 2million open files. Just want your opinion on these settings
I am on ALMA Linux but it is essentially the same on other Linux servers
Many thanks