Issue Let's encrypt - unable to renew *some* subdomains

[Failure404]

I am using a domain with ten subdomains. They all use LE certificates. Automatic certificate renewal was working flawlessly in the past year.
All of a sudden I am getting notified that four of those subdomains can't be renewed due to:

Error: Could not issue a Let's Encrypt SSL/TLS certificate for sub.domain.com. Authorization for the domain failed.
Invalid response from https://acme-v01.api.letsencrypt.org/acme/authz/abcHJ4zObvwxjr-yYkTZoUEshLIrl6H1a6J7rRdUXYZ.
Details:
Type: urn:acme:error:connection
Status: 400
Detail: Fetching http://sub.domain.com/.well-known/acme-challenge/AB_C4vXyP4nodaL0tBTo3IhWk1mkheAAqozYZORXw0s: Connection refused

It drives me crazy. My other subdomains still renew fine, so I don't understand why these particular subdomains get "connection refused".
I tried alot, I even completely removed a subdomain and recreated it just using default settings - connection refused. The above error message is all I get - absolutely nothing else in the logs. Speaking of logs, I even don't see those queries by LE when another subdomain gets successfully renewed. I am sure those appeared a few months back. I also temporarily disabled the firewall which didn't change anything. The URL where LE gets connection refused works fine in my browser.
I am using Plesk Onyx 17.8.11 Update #47 on Debian 8.11.
Any help would be greatly appreciated since I am completely lost here.
Thanks!

 

[Bitpalast]

What is happening in the logs and in the browser when you directly request
http://sub.domain.org/.well-known/acme-challenge/AB_C4vXyP4nodaL0tBTo3IhWk1mkheAAqozYZORXw0s
(with your real subdomain as the domain name)?

 

[Failure404]

Thank you kindly for your reply.
When I open the connection refused URL (/.well-known/acme-challenge/kpzyzJGoVnVyXaWzW4YDPz2TMxz15MyeGGBXmROYcvo) in my browser, I can see text:

kpzyzJGoVnVyXaWzW4YDPz2TMxz15MyeGGBXmROYcvo.Fb0wy3WjW0qlYIrj3AFkchyg26updtTNhADTRmoGZto

I can see my own request in the log:

2019-03-31 12:52:35 Access xxx.xxx.xxx.xxx 200 GET /.well-known/acme-challenge/kpzyzJGoVnVyXaWzW4YDPz2TMxz15MyeGGBXmROYcvo HTTP/1.1

I don't see any request by LE, also when successfully renewing another sub domain.

 

[Bitpalast]

The subdomain has probably an IPV6 record? Please remove that record from DNS, wait a few hours, then try again.

 

[Failure404]

I am using an external DNS service and my subdomains are all configured the same way.
There aren't any AAAA records. The server uses IPv4 only. Plesk DNS, even though not being used, does not show any AAAA records.
"dig sub.domain.com AAAA" does not return any records.

 

[Failure404]

"Connection refused" suggests that it is a connection problem, but could this also mean other things, i.e. file permission problems?
I really don't understand why six subdomains renew successfully, and four others under the same domain don't.
I recently tried a docker proxy rule with one of these four subdomains, but removed it again minutes later. I've read that docker proxy rules break LE auto renewal.
Also, here is a guy with a similar problem, but I don't know if he resolved it:
Troubleshooting failed certificate installation in Plesk Let's Encrypt extension

 

[Bitpalast]

Have you activated ECDSA certificate type in the panel.ini configuration file? If so, remove that, then remove the existing certificates and create new certificates for the subdomains. We have had several such "random" issues while we were trying that algorithm. Once removed, things worked again.

 

[Failure404]

I doublechecked and its set to RSA.
Also, during my troubleshooting process I recently tried obtaining a wild card certificate for my main domain and therefore added

[ext-letsencrypt]
acme-directory-url = "https://acme-v02.api.letsencrypt.org/directory"
acme-protocol-version = "acme-v02"

After this, I got the same connection refused error for my main domain and all previously still working subdomains.
Reverting this gets me back to square one, main domain and six other subdomains renew fine, four other don't.

 

[Bitpalast]

I can only think of any kind of URL rewriting or redirect taking place. If not in the .htaccess file, then maybe in the Apache or nginx settings?

 

[Failure404]

I've tried alot, at the end I even removed a subdomain - expecting to get any exisiting cfg files removed - and then recreating it from scratch without touching any default setting.
I have manually created a 777 /.well-known/acme-challenge/ folder, I have removed all files in /usr/local/psa/var/modules/letsencrypt/etc/live/sub.domain.com
The default nginx.conf has this for both 80 and 443:

#extension letsencrypt begin
location /.well-known/acme-challenge/ {
root /var/www/vhosts/default/htdocs;

types { }
default_type text/plain;

satisfy any;
auth_basic off;
allow all;

location ~ ^/\.well-known/acme-challenge.*/\. {
deny all;
}
}
#extension letsencrypt end

I tried changing root to /var/www/vhosts/domain.com/sub.domain.com because there a /.well-known/acme-challenge gets created for some reason, but that didn't help.

 

[Bitpalast]

Are you using a distributed/cached DNS/CDN service? Cloudflare? Could it be possible that depending on the requesting party that service delivers a different response to the party than what your server would deliver if it was asked directly?

 

[Failure404]

No, it's a very basic DNS service.

I am trying to find any difference between subdomains where renewal is working and those where it doesn't.

Working subdomain:
No /.well-known/acme-challenge/ gets created at /var/www/vhosts/domain.com/sub.domain.com during renewal, only /var/www/vhosts/default/htdocs/.well-known/acme-challenge/ is used. (as specified in nginx.conf by LE extension)

Non working subdomain:
/var/www/vhosts/domain.com/sub.domain.com/.well-known/acme-challenge/ gets created with the needed file inside AND /var/www/vhosts/default/htdocs/.well-known/acme-challenge/ is used with the needed file inside.

Why this happens is beyond me, since in both cases nginx.conf specifies /var/www/vhosts/default/htdocs as root for /.well-known/acme-challenge/ and nothing else is configured. This happens both with a subdomain created from scratch and existing subdomains which renewed fine over the past year and now fail.
I don't know if this is part of the problem, but this is the only difference I could find so far.

 

[Bitpalast]

I think there was a discussion somewhere to save the token files outside the website directory, but I don't recall where I read that. Maybe in a newer Plesk version??? I just don't remember. This seems to be a question for @Ruslan Kosolapov or @custer or for official Plesk support.

 

[regisg]

The subdomain has probably an IPV6 record? Please remove that record from DNS, wait a few hours, then try again.

I had the same issue and it has been resolved thx to this solution : actually, I only had to unselect the IP V6 adress in the Plesk domain configuration : 2019-04-10_12h56_01.jpg
I also had to uncheck the 301 redirect from HTTP to HTTPS in the Plesk domain configuration : 2019-04-10_20h50_21.jpg

 

[GravuTrad]

only unselect ipv6 is sufficient. plesk must find the problem with ipv6 configuration.

 

[ChristophRo]

Has not really anything to do with Plesk, as this one is completely on LetsEncrypt and their http validation service. (unfortunately they seem to not care that their system is broken, when it comes to IPv6)
We have/had the very same problem on non Plesk Linux and Windows servers - on Windows we even use different software (ACMESharp and ZeroSSL) to issue and update LetsEncrypt certificates.

In the end, you will currently always run into problems if:

a) there is a 301/302 redirect happening for the validation url (http://yourdomain.tld/.well-known/acme-challenge/xxxx)
b) your url/website is reachable via IPv6

As we can't change A) with Plesk at the moment, the only way to go is remove the AAAA record

 

[learning_curve]

....In the end, you will currently always run into problems if:
a) there is a 301/302 redirect happening for the validation url (http://yourdomain.tld/.well-known/acme-challenge/xxxx)
b) your url/website is reachable via IPv6
As we can't change A) with Plesk at the moment, the only way to go is remove the AAAA record

Just for information on this subject;

We have IPv6 enabled on all of our domains (with enabled AAAA DNS records) they all have Plesk's 'Permanent SEO-safe 301 redirect from HTTP to HTTPS' setting enabled and they all use Let's Encrypt Certificates.

For all domains that have no sub-domains, we use the Plesk Let's Encrypt Extension, as this provides auto-renewal Let's Encrypt Certificates and we've never had any issues.

For all domains that do have sub-domains (so we need * wildcard certificates) and/or if we use multi-domain certificates, we manually renew all these Let's Encrypt Certificates, using ACME SH but again, we've never had any issues.

 

[ChristophRo]

@learning_curve

Are you using nginx, or maybe apache2 only?
Cause it does only redirect the .well-known/acme-challenge requests to https and/or primary domain, when using nginx as frontend proxy

 

[learning_curve]

Are you using nginx, or maybe apache2 only?
Cause it does only redirect the .well-known/acme-challenge requests to https and/or primary domain, when using nginx as frontend proxy

Good call and our answer is no. We have no domains (currently) that setup to use either Apache or Nginx only.

All the domains / sub-domains / Plesk host domains etc that we mentioned above, have Nginx frontend proxy / Smart static files processing enabled through Plesk.

Again for information, some of those domains / sub-domains do use applications that are 100% dependent on Apache (no specific Nginx coding at all, is provided with them and there's lots and lots of .htaccess files etc) but to date, these all run perfectly with the Plesk Nginx frontend proxy / Smart static files processing settings enabled.

 

[GravuTrad]

Has not really anything to do with Plesk, as this one is completely on LetsEncrypt and their http validation service. (unfortunately they seem to not care that their system is broken, when it comes to IPv6)
We have/had the very same problem on non Plesk Linux and Windows servers - on Windows we even use different software (ACMESharp and ZeroSSL) to issue and update LetsEncrypt certificates.

In the end, you will currently always run into problems if:

a) there is a 301/302 redirect happening for the validation url (http://yourdomain.tld/.well-known/acme-challenge/xxxx)
b) your url/website is reachable via IPv6

As we can't change A) with Plesk at the moment, the only way to go is remove the AAAA record

Are you sure of this? cause they said it's a plesk problem so if each part resend the ball, we can only do a match...