Question Moodle installation on plesk / firewall configuration

[Coltrane70]

Dear community,

we keep having serious problems with our moodle site running on a root server with Debian 10.9 and Plesk Obsidian 18.0.35.

We've got a single subscription we host ourselves. It's a moodle server 3.10.4 (Build: 20210510) (2020110904) .

Our problem seems to be that almost all the available memory (64 GiB) is used as cached memory (memory:memory-cached:value 60 GiB).

When I check the moodle status report it says that there are 437 tasks in the adhoc queue.

Here's some information on the site:

PHP version used: 7.4.18, run PHP as FastCGI application served by Apache.
memory_limit set to 1024 M

The problem is that the site often hangs for some time and sometimes even crashed so that we have to restart the server.

An ideas on what the reason could be?

 

[Bitpalast]

RAM cache in Linux: It is correct and the expected behavior that a lot of RAM is used for caching. Linux will free these portions as needed by other processes. There is nothing wrong with this.

PHP as FastCGI: This means that each script will start its own additional PHP instance through the FastCGI interface. On systems with lots of traffic this causes unnecessary RAM and CPU load. Instead, switch the handler to PHP-FPM served by Apache (if you need .htaccess configurations) or better PHP-FPM served by Nginx (if you can translate your .htaccess rules into Nginx configuration directives).

Site hangs: The reason can normally be identified through either the webserver error_log (Icon "Logs") or the PHP-FPM logs (once switched to PHP-FPM).

Swap: This is mentioned in your headline, but no more details have been given in the description. If the system is using a lot of swap space in addition to RAM, it can be an indication that your RAM is too small for what the software needs.

 

[Coltrane70]

Dear Peter,
Thank you for your reply!

RAM cache in Linux: It is correct and the expected behavior that a lot of RAM is used for caching. Linux will free these portions as needed by other processes. There is nothing wrong with this.

As for our memory usage and swap files the output in the advanced monitoring of plesk looks like this:



There's no swapping whatsoever. As I thought that all the memory is being used by a process that causes our site to hang, I assumed that this may be due to the fact that there's no swapping. We've got 64 GiB of RAM available. This is what memory usage looked like with only two people online. (At about 11.00 am I had restarted the server.)

PHP as FastCGI: This means that each script will start its own additional PHP instance through the FastCGI interface. On systems with lots of traffic this causes unnecessary RAM and CPU load. Instead, switch the handler to PHP-FPM served by Apache (if you need .htaccess configurations) or better PHP-FPM served by Nginx (if you can translate your .htaccess rules into Nginx configuration directives).

You suggest changing the PHP handler. I tried this yesterday, but as soon as I switched thesite looked broken. Do I have to do anything else after switching the handler?

Site hangs: The reason can normally be identified through either the webserver error_log (Icon "Logs") or the PHP-FPM logs (once switched to PHP-FPM).

I've raised the debugging level inside moodle, but still haven't found the error_log where the eroor messages are saved. Or are we talking about the error_log of our plesk, because I just noticed on the phpinfo() page of our plesk that the value for error_log is empty.

Swap: This is mentioned in your headline, but no more details have been given in the description. If the system is using a lot of swap space in addition to RAM, it can be an indication that your RAM is too small for what the software needs.

(see above)


We noticed that the problems had gone after we had switched off the plesk firewall. In the moodle forum someone suggested I would have to add the relevant moodle directories to the openbase_dir php settings like:

{/}{:}/var/www/vhosts/sampledomain.com/httpdocs
{/}{:}/var/www/vhosts/sampledomain.com/moodledata

But that doesn't seem to make sense, as the directories are within the plesk subscription.

The problems I'm referring to were that we got "Server Error 403 Forbidden - You do not have permission to access this document" messages when performing administrative tasks in our moodle. As the server hung for some time after these messages and had to be restarted several times, we thought that this might be due to the assumed memory problem mentioned above. Except for that the site seems to be working ok.

 

[Bitpalast]

RAM usage: It's o.k., no need to worry. It's the expected Linux cache behavior. Your system does not use disk swap, so it does not actually require more RAM.

Reason for the "outages" of your website: Your Moodle is causing 403 errors that show up in your error_log (of the web server, icon "Logs" in "Websites & Domains") as "client denied by server configuration". These entries are evaluated by Fail2Ban ("Tools & Settings" > "IP Address Banning"). Once the threshold of these errors is reached, the Apache jails is triggered and your IP is being jailed for t seconds as configured in "IP Address Banning". You gain access to the site again after a server restarted, because iptables is cleared upon a restart.

Solution: Find out what exactly Moodle is accessing when it throws the "client denied by server configuration" error and make these resources available. There are some articles on that in Google, e.g. search for "Moodle 403" and also lots of articles on "Moodle client denied by server configuration". Quick & dirty "solution": Disconnect your DSL router from the Internet, wait a few minutes, reconnect to obtain a new dynamic IP address from your ISP or disconnect your smartphone from WIFI, then use your smartphone to login to Plesk and remove the IP of your DSL connection from the Fail2Ban block list.

PHP-FPM: It is thinkable that the service is not running It should auto-start when a site is switched to it, but maybe it does not for some awkward reason. On the Linux console you can
# service plesk-php74-fpm status
to see if the service is running (for other PHP versions: ...php73..., ...php72... etc.).

 

[Coltrane70]

RAM usage: It's o.k., no need to worry. It's the expected Linux cache behavior. Your system does not use disk swap, so it does not actually require more RAM.

Reason for the "outages" of your website: Your Moodle is causing 403 errors that show up in your error_log (of the web server, icon "Logs" in "Websites & Domains") as "client denied by server configuration". These entries are evaluated by Fail2Ban ("Tools & Settings" > "IP Address Banning"). Once the threshold of these errors is reached, the Apache jails is triggered and your IP is being jailed for t seconds as configured in "IP Address Banning". You gain access to the site again after a server restarted, because iptables is cleared upon a restart.

Solution: Find out what exactly Moodle is accessing when it throws the "client denied by server configuration" error and make these resources available. There are some articles on that in Google, e.g. search for "Moodle 403" and also lots of articles on "Moodle client denied by server configuration". Quick & dirty "solution": Disconnect your DSL router from the Internet, wait a few minutes, reconnect to obtain a new dynamic IP address from your ISP or disconnect your smartphone from WIFI, then use your smartphone to login to Plesk and remove the IP of your DSL connection from the Fail2Ban block list.

PHP-FPM: It is thinkable that the service is not running It should auto-start when a site is switched to it, but maybe it does not for some awkward reason. On the Linux console you can
# service status plesk-php74-fpm status
to see if the service is running (for other PHP versions: ...php73..., ...php72... etc.).

Hello Peter,
thank you so much for this really plausible explanation! If I got you correctly, there wasn't really a server outage but my IP address got banned by our server as I was the one causing the 403s.

here's the log file with the apache error referring to the Fail2Ban reaction:




In order for unoconv and ghostscript to work I've added some entries to the open_basedir setting in our PHP configuration. Are they the ones causing the problem? Or is it rather the following: "File(/proc/loadavg) is not within the allowed path(s)" ?

 

[Bitpalast]

Teh open_basedir settings are irrelevant in this case.

Instead, the website is offending a Comodo rule. There is an alternative solution that you can try: You see the [id "..."] part in the error messages. Take the ID number from there and add it as an exception to the "Web Application Firewall". Do this for all IDs that are associated with 403 error messages. This will keep ModSecurity from reacting on the specific cases, hence the Fail2Ban jail won't be triggered.

 

[Coltrane70]

Thank you for analysing the error log. I'll try adding the exceptions.

By the way, when I use your suggested command to check for the PHP handler returns the following result:
# service status plesk-php74-fpm status
# status: unrecognized service

 

[Bitpalast]

Maybe the service has never been installed? Go to "Tools & Settings" > "General Settings" > "PHP Settings" to see which PHP handlers are known to your system. Use Plesk component updater to remove or to add components such as specific PHP versions.

 

[Coltrane70]

Teh open_basedir settings are irrelevant in this case.

Instead, the website is offending a Comodo rule. There is an alternative solution that you can try: You see the [id "..."] part in the error messages. Take the ID number from there and add it as an exception to the "Web Application Firewall". Do this for all IDs that are associated with 403 error messages. This will keep ModSecurity from reacting on the specific cases, hence the Fail2Ban jail won't be triggered.

Adding the IDs did the trick!

Would it be a violoation of security policy if I posted the IDs here, so that other people who want to use moodle with the web application firewall activated? Or would you consider this firewall redundant?

P.S. I also added the IDs in the modsecurity settings in the section "tools & settings" but I guess that this is the superior instance for ModSecurity. Or does that interfere with the individual security settings in the "website & domain" settings?

 

[Coltrane70]

Maybe the service has never been installed? Go to "Tools & Settings" > "General Settings" > "PHP Settings" to see which PHP handlers are known to your system. Use Plesk component updater to remove or to add components such as specific PHP versions.

Obviously it's installed?!?

 

[Bitpalast]

Thank you for analysing the error log. I'll try adding the exceptions.

By the way, when I use your suggested command to check for the PHP handler returns the following result:
# service status plesk-php74-fpm status
# status: unrecognized service

Sorry, I overlooked the additional "status". The correct command would be
# service plesk-php74-fpm status

 

[weltonw]

Quick note @Coltrane70 for future reference - you can just whitelist your IP by adding it to f2b via "Trusted IPs" in Plesk and it won't ban you.

Would it be a violoation of security policy if I posted the IDs here, so that other people who want to use moodle with the web application firewall activated? Or would you consider this firewall redundant?

Not redundant - you can post the ID.

P.S. I also added the IDs in the modsecurity settings in the section "tools & settings" but I guess that this is the superior instance for ModSecurity. Or does that interfere with the individual security settings in the "website & domain" settings?

It doesn't - both are used. One is applied per-site, another globally.

 

[Coltrane70]

Sorry, I overlooked the additional "status". The correct command would be
# service plesk-php74-fpm status

With PHP-FPM installed, what is there to do once you changed the handler to prevent the site from looking all jumbled? When I tried this I got lots of server error messages.

 

[Coltrane70]

Quick note @Coltrane70 for future reference - you can just whitelist your IP by adding it to f2b via "Trusted IPs" in Plesk and it won't ban you.

Thanks John for the advice. But doesn't the modem's IP address change once in a while on behalf of the ISP? Wouldn't I need a permanent IP address?

Not redundant - you can post the ID.

OK, so for all the people running moodle in a plesk subscription with the web application firewall (ModSecurity) installed, up till now we had to add the following IDs to the exception rules to prevent 403 server messages "access denied":

210492
210730
211540
214420
214940
240210
242994

It doesn't - both are used. One is applied per-site, another globally.

 

[weltonw]

> Thanks John for the advice. But doesn't the modem's IP address change once in a while on behalf of the ISP? Wouldn't I need a permanent IP address?

It may, it may not. Totally depends on your ISP. We firewall off everything so only internal networks (ie, VPN) can access the server, so we just whitelist the gateway IPs. You could whitelist your modem IP - it usually does not change very often.

 

[Bitpalast]

With PHP-FPM installed, what is there to do once you changed the handler to prevent the site from looking all jumbled? When I tried this I got lots of server error messages.

Can you please post the specific error messages here?

 

[Coltrane70]

Can you please post the specific error messages here?

I've tried to change the settings again and the site looked broken again. Here's the corresponding error message.

 

[Bitpalast]

This seems to be a very specific Moodle-related issue. There are several Google results on "invalidargorconf". I am afraid that in terms of Plesk configuration I cannot assist you with this. I'd guess it has something to do with a path setup, e.g. a temporary path that might need to be changed to reside inside your subscription rather than anywhere on the server. But that's only a guess by experience. Is this the only error message that is thrown?

 

[Coltrane70]

This seems to be a very specific Moodle-related issue. There are several Google results on "invalidargorconf". I am afraid that in terms of Plesk configuration I cannot assist you with this. I'd guess it has something to do with a path setup, e.g. a temporary path that might need to be changed to reside inside your subscription rather than anywhere on the server. But that's only a guess by experience. Is this the only error message that is thrown?

No, there are plenty of error messages referring to the theme plugin we have installed. So I guess that your assumption with a path setup being wrong when the php handler is changed is correct. Here's an example:



These are only the first of a large number of errors referring to our moodle theme, which is the one called 'lambda'.

 

[Bitpalast]

The 404 is just a "not found". Are you using PHP-FPM via Apache or PHP-FPM via Nginx? Are you using Nginx caching? It is thinkable that when all data is being processed through Nginx only, rewrite rules are missing. In that case, try to disable Nginx caching, make sure that Nginx does not handle .js files (set the option in the Apache & Nginx configuration icon).

Some other users with a similar issue here have reported that a restart of Nginx and Apache solved their issue.

If you suspect that the theme is the culprit, try to use a default theme to rule out issues with the theme.

It's not really a Plesk or web server issue.