Issue Let's Encrypt "urn:ietf:params:acme:error:caa" 403 failure

[iainh]

Server operating system version

CentOS Linux 7.9.2009 (Core)

Plesk version and microupdate number

Plesk Obsidian Version 18.0.54 Update #4, last updated on Aug 25, 2023 03:44 AM

There seems to be numerous items about:

Details Invalid response from https://acme-v02.api.letsencrypt.org/acme/finalize/356300830/211673632166. Details: Type: urn:ietf:params:acme:error:caa Status: 403 Detail: Error finalizing order :: Rechecking CAA for "example.com" and 18 more identifiers failed. Refer to sub-problems for more information

Sadly, despite lots of investigation, none seem to resolve my issue. A few points...

1. I am starting to see this across more than one subscription
2. None of the subscriptions/Plesk configurations are new and there have been no changes
3. All these subscriptions have bee auto-updating happily for years, with only the occasional LE request to update the <_acme-challenge> string/value
4. The subscriptions are typically redirection services, configured via Plesk and so do not serve HTTP content other than 301 redirect responses
5. None of the domains in these subscription groups have or have ever had CAA records
6. There are therefore no CAA restrictions and LE has, as mentioned, been happily provisioning the various domains included in these subscriptions (via Plesk) for years
7. The subscriptions are using wildcard certs to redirect any FQDN to the target location
8. Recently I am seeing this CAA 403 (Forbidden) error and have yet to find the solution as to a request for what is being forbidden

Interestingly, in the subscription I am looking at at the moment, there are 15 domains, the main subscription domain + 14 others. In the other, there is the principal domain + 5 others. The errors through read:

For the 1 + 14 domains subscription:
Detail: Error finalizing order :: Rechecking CAA for "<domain-name>" and 18 more identifiers failed. Refer to sub-problems for more information

For the 1 + 5 domain subscription:
Detail: Error finalizing order :: Rechecking CAA for "<domain-name>" and 11 more identifiers failed. Refer to sub-problems for more information

Both are using wildcard and so I'd expect a single wildcard cert per domain, so what are these "identifiers" where there are more "identifiers" than domains/FQDN to check? What request is generating the 403?

It would help if there was more information and if there was some pointer to the mentioned 'sub-problems' for the additional information mentioned. What additional information? Where? How do I find the transcript of what's going on and which request is resulting in the 403? How do I diagnose and so resolve what's going on, and why the change of behaviour in subscriptions that have been in place years?

 

[Peter Debik]

It seems that we have not seen this issue. Are you using Cloudflare or another proxy service in front of your server? Are DNS records managed on your Plesk server, did you verify that they are really managed by that server, e.g. by checking which nameservers are registered in the domain names?

 

[iainh]

Just to add to my own thread...

• I took the smaller 1 + 5 domain subscription
• I de-selected all the additional domains and tried again ... it still failed
• I then de-selected wildcard and when to domain and www.domain options only ... it worked! ... however...

While the 'domain + www only' v wildcard worked, Plesk did report:

Encountered issues while issuing the certificate for <domain>​

Details​

Failed to retrieve authorization for '<domain>'​

But the domain reports as being good to 28-Dec-23 despite failing to retrieve authorisation!

Checking the cert, I can see...

• Yes, good to 28-Dec-23
• The principal subscription domain is included as a wildcard
• The other domains are included unqualified + www

 

[iainh]

It seems that we have not seen this issue. Are you using Cloudflare or another proxy service in front of your server? Are DNS records managed on your Plesk server, did you verify that they are really managed by that server, e.g. by checking which nameservers are registered in the domain names?

Hi Peter, no, nothing in front and Nginx disabled just to rule that out.

No, DNS is not managed by the Plesk service and yes I know where the autorotative DNS are and it all been working for years. Yes I have checked the DNS for all names and one idea I followed was a misreading of an wildcard SPF by LE, but that would require LE to read TXT not CAA records.

You can see one of the DNS checks at:

DigWebInterface.com

Extensive web interface to dig

www.digwebinterface.com

Note the first record is the subscription principal domain, all the others 'courtesy' names which return the wildcard SPF (to prevent snowshoe attacks). If you check CAA you'll see all zones have no CAA, and why check TXT? Well, it shows all zones are active and responding

 

[Peter Debik]

I wonder if it could be an issue on Let's Encrypt side, because the 403 makes little sense if it's a wildcard certificate that checks the token via DNS. 403 means that the service can't access files of your domain, but it should not at all try to check them there. For that reason I wondered whether you have a proxy in front of the server. I guess I cannot help here. Your best bet could be to ask Plesk support https://support.plesk.com by a ticket, so that they can try to check it diredctly on your server.

 

[iainh]

Many thanks Peter.
Yes, I understand a 403, but it's not just "can't access", but is being forbidden .. it's a 403, not a 404. I'll have a look at the logs and maybe put a tail on while an attempt to reissue is running in the hope of seeing what it is that's being attempted and denied.

 

[iainh]

Just to add to the thread, but not expecting a magic answer, other subscriptions on the same host all update okay, even now, one that was suffering the same 'urn:ietfarams:acme:error:caa' 403 error. Trying to work out what is the key issue, although I cannot see why one that was failing, is now working. That's the worst kind of 'fix'!

 

[iainh]

This is maybe relevant...

Type: urn:ietfarams:acme:error:caa​

Status: 403
Detail: Error finalizing order :: Rechecking CAA for "www.sprakekingsley.org.uk" and 14 more identifiers failed. Refer to sub-problems for more information​

Why relevant? Well, the request is for the root and wildcard, so why would a check be made on a specific qualified name like 'www' when a wildcard is (or should be) being requested? And selecting only to renew the root name, i.e. deselecting both wildcard and 'Include www' works! The cert renewal goes through okay. So there's nothing inherent in the server setup that's stopping things, it's something to do with a CAA check on the 'www' name.

Going back now to see if I can issue with wildcard selected, some odd behaviour from Plesk. It stopped with the _acme.challenge. update message ... only the value it provided was the value already visible in public DNS? Why ask for this value update when this is the published value? And the renewal failed ... again. Interestingly, each time it reports this CAA 403 failure, it's a different alternate name being quoted "and 15 (or other number) other identifiers", and... it's always a www name. Different domain name, but always the www-qualified name that's quoted as the CAA check failure.

And now for the real kicker...

I went to SSLLabs to have a general look at the server setup and cert (fun in a browser as all the names are on 301 redirect), and... the cert is in place and all names are included! Yes, despite the CAA 403 error, the cert is there and includes:

1. The unqualified principal domain
2. The wildcard for the principal domain
3. The unqualified names for all the 'domain aliases'
4. The 'www' qualified name for all the domain aliases

So is the report of the error the only issue? Guess I'll need for this to come around again, then once I see the warning from Plesk about the cert coming up for expiry (meaning it's not auto-updated), try a manual refresh (in case of _acme.challenge. update being required), and if I see this CAA 403 error, ignore it and just examine the cert and see whether it's actually been issued as seems to be the case here.

Very odd behaviour!

 

[iainh]

This issue is back and looking for anyone reporting this issue I find my own post

What I notice is that I mention two subscriptions having this 'checking CAA' error ... both for domains that have no CAA and never have. I also see I said...

• I took the smaller 1 + 5 domain subscription
• I de-selected all the additional domains and tried again ... it still failed
• I then de-selected wildcard and all the domain aliases and only selected the principal domain and 'www' qualified name options ... and it worked! ... however...

...that despite requesting only the principal name and 'www' name, the new certificate included:

• The principal subscription domain is included as a wildcard
• The other domains are included with both unqualified + www names

i.e. more than the request which was for principal domain unqualified and 'www' names

Well, the same 1+5 subscription is back to haunt me as 28-Dec is coming due.

The 'getting more than I requested' is of interest as I have again deselected all but the principal domain, but the LE error reports include some of the de-selected domain alias names:

Could not issue an SSL/TLS certificate for example.com
Details
Could not issue a Let's Encrypt SSL/TLS certificate for example.com.

Details
Invalid response from https://acme-v02.api.letsencrypt.org/acme/finalize/356300830/227970812736.

Details:
Type: urn:ietfarams:acme:error:caa

Status: 403
Detail: Error finalizing order :: Rechecking CAA for "www.example-alias.com" and 10 more identifiers failed. Refer to sub-problems for more information

Re the 403 Forbidden, note this is an invalid response from https://acme-v02.api.letsencrypt.org/acme/finalize/356300830/227970812736
i.e. it is LE itself that seems to be returning the 403, not my server responding to a request from LE.

I guess that means I need to go ask LE. If I get any sense, I'll report it here. It may help others ... or me in the future looking up the same issue again

 

[mow]

Have you tried just setting that CAA record? It's best practice anyway.

 

[iainh]

I have been trying all manner of things @mow What's more, I have some interesting findings and questions for Plesk...

I have indeed set CAA for the various domains, however, and yes as a CISO I recognise you 'best practice' point, CAA are not required and both these 'problem subscriptions' have worked and others still do without CAA RR. However, creating CAA RR does not solve the problem. Read on...

This is where this becomes a query to Plesk. What I have found and documented is that:

• If I leave the renewal options as principal domain + wildcard + domain aliases with unqualified and www names, then the renewal fails with the LE "Rechecking CAA" error
• If I deselect all options, i.e. deselect principal name wildcard and www and deselect all domain aliases, the renewal works and the cert is renewed with the full load of principal domain + wildcard + all domain aliases with unqualified and www names

Note this carefully: Requesting 'a full load' renewal fails with a CAA check error, irrespective of the CAA RR in place, while deselecting everything gives a 'full load' cert rotation, thereby proving there is no issue with the CAA RR.

I am picking up with LE about why they misdirect in their reports, as the 'full load' renewal can work proving all CAA are fine ... and in the past this has included there being no CAA RR ... so, why are they reporting this as the issue? But from a Plesk perspective, the interesting question is what is being requested, what logging they might be or which could be enabled, and how it is when deselecting all renewal options I actually get a 'full load' cert update. What is different in the two requests, one of which works, the other fails, but the 'full load' of names is issued even when not requested.

As an example look at https://www.ssllabs.com/ssltest/analyze.html?d=sprakekingsleyllp.co.uk and note the SAN list. This renewal worked by requesting via the UI, only the unqualified principal domain name, as below. Explain that! Would love to see the actual LE request and responses to work out what's going on.