Resolved Magento 2.4.2, Varnish in Docker and Getting 501 PURGE errors in NGINX

[wjpbill]

Server operating system version

Ubuntu 20.04.6 LTS

Plesk version and microupdate number

18.0.56

Hello

I'm running Plesk Obsidian Version 18.0.56 and Magento 2.4.2

I've followed these instructions to get Varnish installed and working Varnish for WordPress in a Docker container and it is working extremely well for serving pages etc.

The problem is that if I try and flush / clear the cache, I get a 501 error in Nginx saying
/2023:12:57:14 +0000] "PURGE / HTTP/1.1" 501 267 "-" "-" –

I've spent days trying to re-configure the purge IP addresses but nothing I do works.

The only way I can clear my Magento 2 cache is to restart the Docker Container but that means everytime that happens, the site grinds until the cache is re-built.

It does have Cloudflare in front of it but as I'm getting a 501 in Nginx, that indicates it is reaching the site but not doing anything.

More detail can be seen here - Varnish Cache won't clear on Magento 2.4.2

I would really really appreciate some help.
Thanks

 

[Peter Debik]

If you are still using Apache on this installation, please try to increase the pm.max_children setting in your PHP-FPM configuration to a high value. "high" ist relative, but it probably needs to be a few dozen or more.

 

[wjpbill]

Hello
Thanks for the reply.
I've increased it to 48 from 20 but still get
[17/Dec/2023:13:49:13 +0000] "PURGE / HTTP/1.1" 501 267 "-" "-"
I think increased it to 96 from 48 and got
[17/Dec/2023:13:51:30 +0000] "PURGE / HTTP/1.1" 501 267 "-" "-"

I thought 501 was telling me that the service isn't available, i.e. Not Implemented?

 

[wjpbill]

I've checked the Apache logs and it's also throwing errors in there at the same time as the NGINX errors

AH00135: Invalid method in request PURGE / HTTP/1.0

The full error is

[Sun Dec 17 14:15:35.698296 2023] [core:error] [pid 2922959:tid 140339591505664] [client 77.68.35.142:0] AH00135: Invalid method in request PURGE / HTTP/1.0

I find it odd however that I'm not calling Port 0. In env.php the port is port 80 not 0

 

[wjpbill]

I'm beginning to wonder if it's because the http_cache_hosts line in env,php in Magento 2, specifies an IP address.
That IP address isn't being processed by Plesk config because there are multiple vhosts on the server so it has no idea what to do with it.
Varnish is running in a Docker container as per Varnish for WordPress in a Docker container but when the server receives the PURGE command on the server IP, it just goes eh? and can't process it.

Does that sound correct?

If that is the case, does that mean running Varnish in Docker in a container isn't going to work and I need to install it as a service?

 

[WebHostingAce]

Please see - Instruction - How to Install Magento 2.4 on Plesk

 

[wjpbill]

Please see - Instruction - How to Install Magento 2.4 on Plesk

Thanks but that doesn't contain any specific instructions for Varnish config.

There is a link to Input - Plesk Magento 2 Required Services in Dockers - Varnish, Redis, Elasticsearch, RabbitMQ which itself has a link to Input - Magento 2 | Plesk | Varnish Setup in a Docker which is basically what I've done but it doesn't work.

Fundementally, you cannot call a vhost by IP if your Plesk installation only has one IP.

If you try and add the server IP to Magento's http_cache_hosts when tries to go to the IP to flush the cache, the vhost it reaches it effectively Plesk control panel and that means that the Docker and domain config for Varnish isn't relevant or reachable.

In those instructions you sent me, they say to add 3 records into the acl purge config
1. The Server/Website IP
2. The Docker IP range
3. 127.0.0.1

1. The Server/website IP fails because of what I've said above
2. I've added the Docker IP range to iptables and allowed it to Port 7080 but Flush doesn't use that from Magento 2, it sends the website IP
3. See point 2

The website is caching perfectly using Varnish in a Docker container as per those instructions but you cannot flush the cache, nor does Magento invalidate it, without restarting the container everytime.

There has to be a way to get Plesk and Varnish to play nicely on a VPS with a single IP.

 

[wjpbill]

Thanks but that doesn't contain any specific instructions for Varnish config.

There is a link to Input - Plesk Magento 2 Required Services in Dockers - Varnish, Redis, Elasticsearch, RabbitMQ which itself has a link to Input - Magento 2 | Plesk | Varnish Setup in a Docker which is basically what I've done but it doesn't work.

Fundementally, you cannot call a vhost by IP if your Plesk installation only has one IP.

If you try and add the server IP to Magento's http_cache_hosts when tries to go to the IP to flush the cache, the vhost it reaches it effectively Plesk control panel and that means that the Docker and domain config for Varnish isn't relevant or reachable.

In those instructions you sent me, they say to add 3 records into the acl purge config
1. The Server/Website IP
2. The Docker IP range
3. 127.0.0.1

1. The Server/website IP fails because of what I've said above
2. I've added the Docker IP range to iptables and allowed it to Port 7080 but Flush doesn't use that from Magento 2, it sends the website IP
3. See point 2

The website is caching perfectly using Varnish in a Docker container as per those instructions but you cannot flush the cache, nor does Magento invalidate it, without restarting the container everytime.

There has to be a way to get Plesk and Varnish to play nicely on a VPS with a single IP.

I have now managed to get Purge to work by setting the website concerned as the default site in Plesk by following this How to access a domain by IP address in Plesk - Support Cases from Plesk Knowledge Base

For the minute, it's the only one using Varnish so not a disaster but that isn't going to work in the New Year.

What it shows is that using the IP to Purge, doesn't work if you have multiple Vhosts on a single IP.

 

[WebHostingAce]

Fundementally, you cannot call a vhost by IP if your Plesk installation only has one IP.

In this case what is the purpose of call a vhost by IP?

If you try and add the server IP to Magento's http_cache_hosts when tries to go to the IP to flush the cache, the vhost it reaches it effectively Plesk control panel and that means that the Docker and domain config for Varnish isn't relevant or reachable.

This is why use different ports.

In those instructions you sent me, they say to add 3 records into the acl purge config
1. The Server/Website IP
2. The Docker IP range
3. 127.0.0.1

Actually its the

2. The Docker IP range

Matches the PURGE request.

The website is caching perfectly using Varnish in a Docker container as per those instructions but you cannot flush the cache, nor does Magento invalidate it, without restarting the container everytime.

I can confirm that instructions should work as it is working across many of our Magento 2 servers.

There has to be a way to get Plesk and Varnish to play nicely on a VPS with a single IP.

Yes, It works with Plesk with a single IP.

 

[wjpbill]

In this case what is the purpose of call a vhost by IP?



This is why use different ports.



Actually its the


Matches the PURGE request.



I can confirm that instructions should work as it is working across many of our Magento 2 servers.



Yes, It works with Plesk with a single IP.

Maybe I'm not explaining myself properly.
Using your instructions, Varnish works as a cache to serve pages, I agree 100%.

However, there is one part that doesn't work and that is purging the cache from the admin or the command line.
The reason it fails is if the default site in plesk is set to any domain in plesk that DOES NOT have the Docker proxy rules configured (don't forget they are domain specific), then the port mapping doesn't work.
That means you either have to specify a domain that does have Docker rules configured or create a Vhost specifically for Varnish that has the rules set up.
If you don't do either of those things and for example, your default domain in Plesk is set to Plesk itself, Nginx will throw a 501 on Purge because it doesn't know what the PURGE command is.

My problem specifically relates to PURGE, nothing else.

 

[wjpbill]

Maybe I'm not explaining myself properly.
Using your instructions, Varnish works as a cache to serve pages, I agree 100%.

However, there is one part that doesn't work and that is purging the cache from the admin or the command line.
The reason it fails is if the default site in plesk is set to any domain in plesk that DOES NOT have the Docker proxy rules configured (don't forget they are domain specific), then the port mapping doesn't work.
That means you either have to specify a domain that does have Docker rules configured or create a Vhost specifically for Varnish that has the rules set up.
If you don't do either of those things and for example, your default domain in Plesk is set to Plesk itself, Nginx will throw a 501 on Purge because it doesn't know what the PURGE command is.

My problem specifically relates to PURGE, nothing else.

That being said, because I've told another website to be default on the IP, I now cannot access the Plesk admin.

 

[WebHostingAce]

However, there is one part that doesn't work and that is purging the cache from the admin or the command line.
The reason it fails is if the default site in plesk is set to any domain in plesk that DOES NOT have the Docker proxy rules configured (don't forget they are domain specific), then the port mapping doesn't work.
That means you either have to specify a domain that does have Docker rules configured or create a Vhost specifically for Varnish that has the rules set up.
If you don't do either of those things and for example, your default domain in Plesk is set to Plesk itself, Nginx will throw a 501 on Purge because it doesn't know what the PURGE command is.

This is completely wrong! Why would you request NGINX to PURGE /? This is nothing to do with NGINX.

If you followed my instructions in the "Input" thread regarding the Varnish Docker for Magento 2,

You do not have to specify http_cache_hosts in env.php. In my instructions, I have mentioned it as optional.

If you do, It should be 127.0.0.1 and the Port 32168. (Varnish Docker is Bind to localhost and the Port is 32168)

 

[wjpbill]

This is completely wrong! Why would you request NGINX to PURGE /? This is nothing to do with NGINX.

If you followed my instructions in the "Input" thread regarding the Varnish Docker for Magento 2,

You do not have to specify http_cache_hosts in env.php. In my instructions, I have mentioned it as optional.

If you do, It should be 127.0.0.1 and the Port 32168. (Varnish Docker is Bind to localhost and the Port is 32168)

Okay, the bit I missed was "Create a directory "/root/varnish/conf" and create a file called "default.vcl" with the above content."


You also confused the issue by adding the link to the post - Varnish for WordPress in a Docker container because those instructions differ from your in the "Input" thread because it doesn't have "Create a directory "/root/varnish/conf" but rather says create a domain specific config which is why PURGE didn't work.

Also, in your "Input" thread, you say it's port 32780 not as in your last reply 32168 which also confused me.

It didn't work until I added 'http_cache_hosts" back into env.php with 127.0.0.1 and the port I've used. I'm not sure why you consider it to be optional but at least it's working now.

I appreciate your help but I didn't particularly like the tone of the last reply, it was quite agressive for no reason.

Thank you for your help, it is now working.

 

[WebHostingAce]

You also confused the issue by adding the link to the post - Varnish for WordPress in a Docker container because those instructions differ from your in the "Input" thread because it doesn't have "Create a directory "/root/varnish/conf" but rather says create a domain specific config which is why PURGE didn't work.

Which part you are confused about? a one click away?

Also, in your "Input" thread, you say it's port 32780 not as in your last reply 32168 which also confused me.

So you are complain about one mistake I have made for the reply?

It didn't work until I added 'http_cache_hosts" back into env.php with 127.0.0.1 and the port I've used. I'm not sure why you consider it to be optional but at least it's working now.

This forum replies are not paid work! Also it should have worked without the 'http_cache_hosts"

I appreciate your help but I didn't particularly like the tone of the last reply, it was quite agressive for no reason.

Just learn how to ask for support from the community in a forum. Dont talk about some things you don't know about like below! These are not requests.. these comments are you are teaching someone about something you dont know!!

However, there is one part that doesn't work and that is purging the cache from the admin or the command line.
The reason it fails is if the default site in plesk is set to any domain in plesk that DOES NOT have the Docker proxy rules configured (don't forget they are domain specific), then the port mapping doesn't work.
That means you either have to specify a domain that does have Docker rules configured or create a Vhost specifically for Varnish that has the rules set up.
If you don't do either of those things and for example, your default domain in Plesk is set to Plesk itself, Nginx will throw a 501 on Purge because it doesn't know what the PURGE command is.

 

[WebHostingAce]

Final words.

You sound like ... [redacted]

 

[wjpbill]

Which part you are confused about? a one click away?



So you are complain about one mistake I have made for the reply?



This forum replies are not paid work! Also it should have worked without the 'http_cache_hosts"


Just learn how to ask for support from the community in a forum. Dont talk about some things you don't know about like below! These are not requests.. these comments are you are teaching someone about something you dont know!!

Final words.

You sound ... [redacted]

It's "You sound" not "You sounds".

There is obviously something wrong with you, it would seem you are having quite an angry day.

Read my reply if you don't understand why I was confused and don't be so condescending.

I don't care if it is paid work or not and I have no idea why that has any bearing on anything.

Teaching is guiding and explaining. You are acting simply as a bully.

I don't appreciate your tone and for that I've reported you.

I hope you have a lovely day and that you calm down.

Merry Christmas

 

[wjpbill]

Final words.

You sound like ...

You can't edit out your bullying. Here's an image of what I got from this guy in email. He calls me [redacted] because I got confused.

[It was edited by the admin, not by the user. And, sorry, I have to edit it again here.]

 

[Peter Debik]

Gee, did this escalate fast ... You guys are keeping me on my toes.

Let's all relax. Forum communication can be challenging, because when you don't see and feel your communication partner, it is easy to become frustrated.

I have removed the not-so-nice comments and words. Let's pause a moment, turn on some Christmas music (or whatever makes a nice relaxing atmosphere), have cookie and a cup of tea and please be friends again.