Issue nginx 504 sporadic / Unknown: open_basedir restriction

[lutsoft]

Hi,

I have just migrated from a CentOS 7 server with CPanel 90 to a CentOS 8 server with Plesk Obsidian. I have used the Plesk Migrator extension to migrate everything. All is running except that almost every day suddenly all connections seem to be unstable. I get nginx 504's, my Plesk license cannot be validated and my mail server is also unreachable. I can see the following error with every request in /var/www/vhosts/:domain:/logs/error_log:

AH01071: Got error 'PHP message: PHP Warning: Unknown: open_basedir restriction in effect. File(/var/cpanel/php/sessions/ea-php72) is not within the allowed path(s): (/var/www/vhosts/lutgens.eu/:/tmp/) in Unknown on line 0'

When I restart the server everything is back to normal. The problem also disapeares on it's own after a while.

Does anyone have a clue how can I fix this and could the open_basedir issue be the reason of the unstable connection?

Regards, Lutsoft

 

[IgorG]

The issue is caused by the fact that the PHP parameter session.save_path is set to the directory /var/cpanel/php/sessions/ea-php72 (either in Domains > domain.com > PHP Settings or in a file .htaccess php.ini, or .user.ini inside website's files), but that directory does not exist on Plesk servers.

To resolve the issue, perform the following actions:

1. Ensure that the parameter Domains > domain.com > PHP Settings > session.save_path is set to its default value /var/lib/php/sessions.
2. Check all files .htaccess, or php.ini, or .user.ini in the website's files and remove or comment out all lines similar to the below one:
session.save_path = "/var/cpanel/php/sessions/ea-php72"
3. Go to Domains > domain.com > PHP Settings and click OK to renew the configuration.

 

[lutsoft]

Hi @IgorG,

Thx for your reply. I found a php.ini in the website root that was generated bij cPanel and was migrated along. In that file was also the line session.save_path = "/var/cpanel/php/sessions/ea-php72".

I have removed the file so that should solve that problem. I hope that this also fixes the erratic connection problems.

 

[lutsoft]

@IgorG Unfortunately the erratic problem still exists. Almost every other day my server somehow hangs and when I look in the logging I almost always see:
fork: retry: Resource temporarily unavailable

In the apache logging:
AH01071: Got error 'PHP message: PHP Warning: mysqli::close(): Couldn't fetch mysqli in
and in nginx logging:
1542#0: *1 connect() failed (111: Connection refused) while connecting to upstream

Today I even could not acces it through ssh anymore. What I already tried and applied the past weeks:

• Increase FastCGI timeout limits set to 120: Website on Plesk server is not accessible: 504 Gateway Time-out: The timeout specified has expired: [client 203.0.113.2:54693] AH01075: Error dispatching request to
• I have set ulimits to 65536: nginx fails to start/reload on a Plesk server: Too many open files
• Also when upgrading to the latest Obsidian v18.0.31_build1800201025.00 os_CentOS 8 I had a lot of errors in the logging with "fork: retry: Resource temporarily unavailable"

I currently have no idea anymore how to solve this...

 

[Bitpalast]

The "open_basedir restriction in effect" issue that you have described first is a totally different issue from a web server or server outage. It has nothing to do with it.

You solved the one, but your current issue is mainly that your system is opening too many files or exceeding inodes or exceeding limits that are set by the virtualization environment provider. For what you are doing on it you simply don't have enough resources.

The solution strongly depends on what operating system you are on and if this is a virtual server or a dedicated server where you have full control over your resources. One thing you'll need to look at is the number of open files on the system (count all open file handles: # lsof | wc -l) and compare them to the limits set in the operating system as defined in /etc/sysctl.conf. It could also be necessary to add "LIMITNOFILE=n" with n=number of max open files you want to allow in usr/lib/systemd/system/nginx.service and /etc/sysconfig/nginx.systemd. I recommend reading a bit online on the open files limit, because there can be several places that you need to update to expand the capabilities of your operating system. Should you be on a virtual server, you may not be able to expand the resources to more than what your virtualization environment is permitting.

 

[lutsoft]

Hi @Peter Debik,

I run CentsOS8 and the server runs on Virtuozzo. I have already set ulimits to 65536.

'lsof | wc -l' gives me 114892 at this moment. And the problem occurs right now. I have now set it to 200000:

'/usr/local/psa/admin/sbin/websrv_ulimits --set 200000 --no-restart'

and the problem is now gone again. How high should I set it? Can I like set it to 9999999 just to be safe?
------------------------------------------------------------------------------------------------------------------------------------------------------------
I just did 'lsof | wc -l' again and I get

-bash: fork: retry: Resource temporarily unavailable 4 times until I get the value. which is 115074 now.

In Plesk I now also get:

Error: The license key is invalid. In order to use Plesk, please obtain and install a new valid license key. The license key is intended only for use with Plesk operating inside the following virtual environments: Virtuozzo containers, VMWare, Microsoft Hyper-V, Xen, KVM, Virtuozzo Server, LXC, Docker.
------------------------------------------------------------------------------------------------------------------------------------------------------------
just did 'grep 'Max open files' /proc/$(cat /var/run/nginx.pid)/limits' again and the value is still 65536. Is it not possible to set it higher?

 

[Bitpalast]

200000 sounds like a good value for your system. You can set it higher if it is reasonable for your system. It depends on many factors what is reasonable, like RAM size, cpu power, disk size etc. Limits like the max open files shall protect your system against accidental overloads. So 9999999 is probably too much. Plesk recommends to set the value at least to 16 x the number of domains you are hosting. It won't hurt to double that to make it 32 times the number of domains.

One thing though: Setting the limit in the web server configurations is insufficient. At least also look into /etc/sysctl.conf, because the default is lower. You should add something like
fs.file-max = 300000
to /etc/sysctl.conf, then reload the system daemon (# sysctl -p). I have chosen 300000 here, because you picked 200000 for the web servers, but there are more services that also open files.

This will still not do the trick. You'll also need to update the settings in /etc/security/limits.conf for the various services. Else you might allow more files for the application software, but your operating system will cap them with its general limits. For example suitable entries could be
nginx soft nofile 200000
nginx hard nofile 200000
root soft nofile 300000
root hard nofile 300000
psaadm soft nofile 200000
psaadm hard nofile 200000
mysql soft nofile 200000
mysql hard nofile 200000
httpd soft nofile 200000
httpd hard nofile 200000
Then again # sysctl -p.

 

[lutsoft]

200000 sounds like a good value for your system. You can set it higher if it is reasonable for your system. It depends on many factors what is reasonable, like RAM size, cpu power, disk size etc. Limits like the max open files shall protect your system against accidental overloads. So 9999999 is probably too much. Plesk recommends to set the value at least to 16 x the number of domains you are hosting. It won't hurt to double that to make it 32 times the number of domains.

One thing though: Setting the limit in the web server configurations is insufficient. At least also look into /etc/sysctl.conf, because the default is lower. You should add something like
fs.file-max = 300000
to /etc/sysctl.conf, then reload the system daemon (# sysctl -p). I have chosen 300000 here, because you picked 200000 for the web servers, but there are more services that also open files.

This will still not do the trick. You'll also need to update the settings in /etc/security/limits.conf for the various services. Else you might allow more files for the application software, but your operating system will cap them with its general limits. For example suitable entries could be
nginx soft nofile 200000
nginx hard nofile 200000
root soft nofile 300000
root hard nofile 300000
psaadm soft nofile 200000
psaadm hard nofile 200000
mysql soft nofile 200000
mysql hard nofile 200000
httpd soft nofile 200000
httpd hard nofile 200000
Then again # sysctl -p.

I added fs.file-max = 300000 and ran # sysctl -p, but I got the message:
setting key "fs.file-max": No such file or directory

 

[Bitpalast]

The "no such file or directory" indicates that in your virtualization this parameter is not globalized. You cannot set it from within the container. This could become a problem. Maybe you can ask your provider to tell you what the maximum number of open files are for your container.

 

[lutsoft]

The "no such file or directory" indicates that in your virtualization this parameter is not globalized. You cannot set it from within the container. This could become a problem. Maybe you can ask your provider to tell you what the maximum number of open files are for your container.

Ok I will ask them tomorrow. Tnx for the support so far!

 

[lutsoft]

Sorry for the late reply. It took a while for my hosting to come with the answer. They say that they have no restriction set on 'Max open files'. After the changes I already made the problem has not occurred until yesterday. Ik haven't got the time yet to look into that further because I have another problem.

Past weekend my home ip address seems to be banned. The connection to my server was unstable and finally I restarted the server. The server was momentarily reachable after it was restarted, but then again not reachable (also not via SSH). I found out that when I use a VPN, or mobile connection I could connect. So somehow my ip is banned somewhere, but I cannot find it. What I tried:

• There is nothing listed in Fail2Ban.
• 'iptables-legacy -L INPUT -v -n | grep "MY_IP"' gives nothing. Only 'Warning: iptables-legacy tables present, use iptables-legacy to see them'. But iptables-legacy is not recognized as a command.

Any ideas where further to look?

 

[lutsoft]

Update: It appeared to be caused by immunify360. I had it installed, but had no license. I purchased one and my ip was on the gray list......