Resolved Lets Encrypt root certificate expiration on 30 September 2021

[TomBoB]

Asking for some guidance and knowledge please. Certificate chains are among the few things I haven't delved into much so far.

The Lets Encrypt root certificate expiration on 30 September 2021. Anyone still running Plesk on CentOS 7 servers are apparently affected as that OS still uses OpenSSL 1.0.2

It appears we need to remove the soon expired root certificate (DST Root CA X3), and add (if not yet in the store) the new ISRG Root X1 self-signed certificate.

This is where I'm not sure, and rather ask here as Plesk might keep certs in different locations then the OS's default. I'm also not sure about the procedure of removing/adding a cert.

Would be great if someone could provide knowledge and guidance on the matter.

Interesting articles:
https://scotthelme.co.uk/lets-encrypt-old-root-expiration/
https://letsencrypt.org/.../dst-root-ca-x3-expiration.../
https://www.openssl.org/.../09/13/LetsEncryptRootCertExpire/

 

[Bitpalast]

@IgorG and @Rasp : What do you think about this? I am not sure what to check or where to look.

 

[IgorG]

Guys, according to DST Root CA X3 Expiration (September 2021) - Let's Encrypt we will issue certificates the same way as before, which will be given to us by the Let's Encrypt service. Everything will be fine, they claim.

 

[Bitpalast]

OK, so from what I understand this is not a server issue, but a client issue for a small set of old clients. When I open a Let's Encrypt secured website like the Plesk panel of any Plesk installation in my browser and check the certificate chain, i can see that the valid, new root certificate is used for validation.

I think the only issue that could possibly be left is when a server system acts as a "browser", e.g. when a file_get_contents() PHP function tries to open a https:// URL and the server does not have the ISRG Root X1 root certificate installed or has the DST Root CA X3 installed, because in that case CentOS 7.x will for example prefer that over the ISRG Root X1 and fail SSL connections.

So I think that as described in

RHEL/CentOS 7 Fix for Let’s Encrypt Change

Take action before September 30, 2021 to avoid problems

blog.devgenius.io

CentOS 7.x servers the DST Root CA X3 certificate needs to be blacklisted. Not for connections of browsers to the server, but for the connections that are server-initiated to other systems.

 

[Kaspar]

@TomBoB thank you for opening this topic. The expiration of the DST Root CA X3 certificate wasn't on my radar. Also @Peter Debik thank you for tagging me.

The methode described in the link Peter posted to blacklist the root certificate worked for me.

 

[TomBoB]

Thanks Peter for providing that insight and the link!! Much appreciated!!

 

[Bitpalast]

O.k., guys, here is an update: CentOS has published an update to the ca-certificates

ca-certificates-2021.2.50-72.el7_9.noarch.rpm CentOS 7 Download

Download ca-certificates-2021.2.50-72.el7_9.noarch.rpm for CentOS 7 from CentOS Updates repository.

centos.pkgs.org

and according to the change log, this removes the expired certificate

2021-09-14 - Bob Relyea <[email protected]> - 2021.2.50-72
- Fix expired certificate.
- Removing:
- # Certificate "DST Root CA X3"

So actually, when installing the upgrade via package manager (it should show up in the Plesk package manager by now), this should also fix the issue.

 

[TomBoB]

Thanks for the update
We kept a close eye on developments and that really seems to sort it.

Me being of the "older semester" - and training and upbringing staff accordingly - I hate it when things you can see from a long way away are being left to the last moment, and then everyone scrambles to make a plan...

Anyway, potential issue averted. And a big thanks to you Peter, and also Igor and Rasp for your input at a point where we were a little lost!!

Cheers,
Tom

 

[exit15]

Guys, all of my websites get the "NET::ERR_CERT_DATE_INVALID" both on Chrome and Edge browsers, Firefox is fine.

OS: Ubuntu 18.04.5 LTS
Product: Plesk Obsidian 18.0.38 Update #2, last updated on Sept 23, 2021 12:18 AM
Check for updates
Checked on Sept 30, 2021 01:35 PM.

OS: CentOS Linux 7.9.2009 (Core)
Product: Plesk Obsidian 18.0.38 Update #2, last updated on Sept 20, 2021 04:03 AM
Check for updates
Checked on Sept 30, 2021 01:49 PM.

I manually renewed the certificates hoping to some miracle, no luck.
Can someone post here specific steps for each OS, I tried to follow that article suggested above, but its too complicated. -- thanks for the help.
p.s. almost all google searches on this topic lead to websites that are affected, what a s**t show.

 

[TomBoB]

use SSH to log into your server.
on the CLI enter the following commands:

# cp -i /etc/pki/tls/certs/ca-bundle.crt ~/ca-bundle.crt-backup

# trust dump --filter "pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10" | openssl x509 | sudo tee /etc/pki/ca-trust/source/blacklist/DST-Root-CA-X3.pem

# sudo update-ca-trust extract

that's for CentOS

do a

# yum update

afterwards for good measure

 

[TomBoB]

It may also be that something on the computer that you use for checking isn't in order. That Firefox works but Chrome and Edge not points in that direction.

 

[Bitpalast]

Yes, this is a browser issue of the user, not a server issue.

 

[Esperio]

yum install ca-certificates did the trick.

 

[exit15]

trust dump --filter "pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10" | openssl x509 | sudo tee /etc/pki/ca-trust/source/blacklist/DST-Root-CA-X3.pem
unable to load certificate
140685354600336:error:0906D06CEM routinesEM_read_bio:no start lineem_lib.c:707:Expecting: TRUSTED CERTIFICATE

Backup worked, but this black listing did not.
Also tried to install and it says I have the latest ones.
This is a problem everywhere when I google it, what a day. While its true that Firefox works, there must be something on the server side we can do. Please help me bypass the above command error. -- thanks.

 

[TomBoB]

I'm assuming you made sure your computers date / time is correct. Otherwise Firefox would also complain.
open chrome. clear all and any browsing and cashing and history. go to website that shows the error you state. hit control-shift-i . click on security tab. click view certificate. click certification path. what / which certs are listed? click on first line, and tell us what it says in view certificate. Click on secnd line and tell us what it says in view certificate.
Otherwise this is pure guesswork and won't get you anywhere.

 

[TomBoB]

also do the following in firefox as well as in Chrome. There'll be differences. Go here and tell us the important details.

 

[Bitpalast]

... there must be something on the server side we can do. ...

@exit15 Again: This is not server issue. Your browsers are lacking the proper root certificate. You need to update the root certificates of your browsers.

 

[Esperio]

@exit15 Again: This is not server issue. Your browsers are lacking the proper root certificate. You need to update the root certificates of your browsers.

Smtp was not working for some of my customers until i installed ca-certificates, so updating thing in server helps to mitigate the issue

 

[Bitpalast]

That's a totally different playground. exit15 had claimed issues when connecting from his or her browsers to not only the websites hosted on Plesk, but many websites using SSL. The reason for that is that in his or her browsers the root certificates are outdated. Esperio seemed to have issues with the mail servers. Sure, when you don't have the root certificates managed on the host system, connecting through SSL won't work correctly for the server. But that's a different issue from what exit15 had described.

 

[matrix]

That's a totally different playground. exit15 had claimed issues when connecting from his or her browsers to not only the websites hosted on Plesk, but many websites using SSL. The reason for that is that in his or her browsers the root certificates are outdated. Esperio seemed to have issues with the mail servers. Sure, when you don't have the root certificates managed on the host system, connecting through SSL won't work correctly for the server. But that's a different issue from what exit15 had described.

It's related to the same issue. Especially Apple devices will throw that error if serverside is not updated. Which is not the case if you are running an old CentOS or Debian version (not 11).

Backup worked, but this black listing did not.
Also tried to install and it says I have the latest ones.
This is a problem everywhere when I google it, what a day. While its true that Firefox works, there must be something on the server side we can do. Please help me bypass the above command error. -- thanks.

For Debian use:
sudo cp /etc/ca-certificates.conf /etc/ca-certificates.conf.orig
sudo nano /etc/ca-certificates.conf
Change “mozilla/DST_Root_CA_X3.crt” in “!mozilla/DST_Root_CA_X3.crt” an save
sudo update-ca-certificates

For CentOS use:
trust dump --filter "pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10" | openssl x509 | sudo tee /etc/pki/ca-trust/source/blacklist/DST-Root-CA-X3.pem
sudo update-ca-trust extract
sudo yum update ca-certificates

If the problem persist, regenerate your certificates and force in your Apache config file for your website the new chain certificate (add SSLCertificateChainFile path to SSL options). You can find it in /opt/psa/var/modules/sslit/etc/live/website/chain.pem or /opt/psa/var/modules/letsencrypt/etc/live/website/chain.pem.