Resolved OCSP stapling with Nginx issue

[Gabor H]

Hi,

Trying to configure SSL cert, also Nginx ssl.conf.
Using Let's encrypt plugin in Plesk to get a free cert, OS is CentOS 7.2
From outside, using SSLlabs, I get A+ rating for the domain, OCSP stapling look line is working.

But when I check Nginx's status, I get the following warning message:

nginx[10840]: nginx: [warn] "ssl_stapling" ignored, issuer certificate not found


Where to fix this issue? SSLlabs gives green mark and YES for OCSP stapling. But why Nginx is alerting?

Regards,
G.

 

[UFHH01]

Hi Gabor H,

if you would like help for your investigations, it's always a good idea to POST the depending configuration files, especially, when you modified this or that.
Using OCSP stapling for example requires additional configuration, which you mentioned, but don't provide as information for investigations.


Consider as well to use this example command from your command line for verification, to be sure that OSCP works as expected for your domain :

echo QUIT | openssl s_client -connect www.YOUR-DOMAIN.COM:443 -status 2> /dev/null | grep -A 17 'OCSP response:' | grep -B 17 'Next Update'

If you get NO output for your command, it means that OSCP stapling doesn't work, else you should get a "OCSP Response Status: successful" response on your command line.

 

[Gabor H]

Hello UFHH01,

Many thanks for your reply!
Of course, I'll post conf files, but didn't want to trash the first post with maybe a useless thing.

So, as I've SNI, then I ran your command adding "-servername www.YOUR-DOMAIN.COM" right after the domain name.
And I got the following response:

OCSP response:
======================================
OCSP Response Data:
OCSP Response Status: successful (0x0)
Response Type: Basic OCSP Response
Version: 1 (0x0)
Responder Id: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
Produced At: Sep 21 12:40:00 2016 GMT
Responses:
Certificate ID:
Hash Algorithm: sha1
Issuer Name Hash: 7EE66AE7729AB3FCF8A220646C16A12D6071085C
Issuer Key Hash: A84A6A63047DDDBAE6D139B7A64565EFF3A8ECA0
Serial Number: 0303E1A40A8F344B3372313F43D9664829C8
Cert Status: good
This Update: Sep 21 12:00:00 2016 GMT
Next Update: Sep 28 12:00:00 2016 GMT


And this is inside the Nginx's ssl.conf:
(/etc/nginx/conf.d/ssl.conf)

ssl_ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384HE-RSA-AES128-GCM-SHA256HE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHAHE-RSA-AES128-SHA256HE-RSA-AES128-SHAHE-RSA-AES256-SHA256HE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHAES-CBC3-SHA:!DSS;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_timeout 60m;
ssl_ecdh_curve secp384r1;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
ssl_buffer_size 4k;
ssl_dhparam /etc/pki/tls/certs/dhparam.pem;
ssl_stapling on;
ssl_stapling_verify on;
add_header Strict-Transport-Security max-age=15768000;


And this is the domain vhost's nginx.conf:
(just the first segment for the cert info)

#ATTENTION!
#
#DO NOT MODIFY THIS FILE BECAUSE IT WAS GENERATED AUTOMATICALLY,
#SO ALL YOUR CHANGES WILL BE LOST THE NEXT TIME THE FILE IS GENERATED.

server {
listen XX.X.XX.XXX:443 ssl http2;

server_name mydomain.com;
server_name www.mydomain.com;
server_name ipv4.mydomain.com;

ssl_certificate /usr/local/psa/var/certificates/cert-h4NlIC;
ssl_certificate_key /usr/local/psa/var/certificates/cert-h4NlIC;
ssl_client_certificate /usr/local/psa/var/certificates/cert-6kfSXy;



Any hint or tips a highly appreciated.


Kind regards,
Gabor

Hi Gabor H,

if you would like help for your investigations, it's always a good idea to POST the depending configuration files, especially, when you modified this or that.
Using OCSP stapling for example requires additional configuration, which you mentioned, but don't provide as information for investigations.


Consider as well to use this example command from your command line for verification, to be sure that OSCP works as expected for your domain :

echo QUIT | openssl s_client -connect www.YOUR-DOMAIN.COM:443 -status 2> /dev/null | grep -A 17 'OCSP response:' | grep -B 17 'Next Update'

If you get NO output for your command, it means that OSCP stapling doesn't work, else you should get a "OCSP Response Status: successful" response on your command line.

 

[UFHH01]

Hi Gabor H,

your configuration for

ssl_trusted_certificate /PATH/TO/YOUR/DOMAIN-SPECIFIC/FULLCHAIN.PEM-FOR-LETS-ENCRYPT/fullchain.pem;

is missing in your domain - specific configuration for nginx.


You are always able to locate domain - specific "Let's encrypt" - files with for example the "locate" - command ( requires "mlocate" to be installed on your server - consider to use the forum - search, if you need more informations to installation and usage of "mlocate" ).

Example:

locate fullchain.pem​

 

[Lloyd_mcse]

Hi guys,
just to add to this, you should add the OCSP Stapling directives on a per domain basis, rather than a server wide, unless all of your domains use Let's Encrypt that is.
I like to create a folder, eg:

/etc/ssl/stapling

Then I create a .pem file for each CA (in this case Lets Encrypt) containing the Intermediate (Let's Encrypt Authority X3) and the Root CA (DST Root CA X3) and upload them to the folder...

/etc/ssl/stapling/Lets-Encrypt-Authority-X1.pem

Once that's done I can go into Plesk...

Domains > domain.tld > Apache & Nginx Settings > Additional nginx directives

Adding...

ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/ssl/stapling/Lets-Encrypt-Authority-X1.pem;
resolver 8.8.4.4 8.8.8.8 valid=300s;
resolver_timeout 10s;

I hope that helps
Kind regards

Lloyd

 

[Gabor H]

Dear UFHH01,

Indeed, that was missing.
Amended nginx configuration accordingly, but still getting the nginx warning:

nginx: [warn] "ssl_stapling" ignored, issuer certificate not found

By the time I found this article about my issue.
It states the following:


(If everything is successful, you will get a result that looks like this:

Output
nginx: [warn] "ssl_stapling" ignored, issuer certificate not found

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Notice the warning in the beginning. As noted earlier, this particular setting throws a warning since our self-signed certificate can't use SSL stapling. This is expected and our server can still encrypt connections correctly.)



So what do you think? May I just simple ignore the warning of Nginx?


Kind regards,
Gabor

Hi Gabor H,

your configuration for

ssl_trusted_certificate /PATH/TO/YOUR/DOMAIN-SPECIFIC/FULLCHAIN.PEM-FOR-LETS-ENCRYPT/fullchain.pem;

is missing in your domain - specific configuration for nginx.


You are always able to locate domain - specific "Let's encrypt" - files with for example the "locate" - command ( requires "mlocate" to be installed on your server - consider to use the forum - search, if you need more informations to installation and usage of "mlocate" ).

Example:

locate fullchain.pem​

 

[UFHH01]

Hi Gabor H,

So what do you think? May I just simple ignore the warning of Nginx?

well, that really depends on you and your domain - visitors, when you use the combination apache+nginx for your domain. If you follow the suggestions, especially the one from @Lloyd_mcse here at your thread, you will notice, that the OSCP - stapling configuration is quite simple and done within seconds. No need to ignore the warnigs, if there aren't any, or what do you think? In addition, if you have let's say "lazy" ( sounds mean, but I just choosed the additional adjective, because they "forgot" to modify for example default FireFox settings => "Use the OSCP to confirm the validity of the certificates" ) site - visitors, it is far better to setup OSCP stapling correctly, to avoid "The page you are trying to view cannot be shown because the authenticity of the received data could not be verified." - messages.

On the other hand, all other browsers than FireFox doesn't have this default setting set and the site - visitors will then never see such a message.

It's really up-to-you, how you decide.

 

[Gabor H]

Hi guys,

Thanks for help for both of you

I think I've found what mistake I've done.
Till this time I used this global Nginx ssl config: /etc/nginx/conf.d/ssl.conf

But few min ago I just found out that there's another Nginx ssl conf here: /var/lib/plesk/ssl_nginx.conf

And this second conf did the magic.

 

[UFHH01]

Hi Gabor H,

I doubt, that "/var/lib/plesk/ssl_nginx.conf" is in any way included ( path is not included anywhere in your nginx - conf - pls. check that again for your domains! ) and therefore, there is no "magic"... sorry to disappoint you.

 

[Walter]

I am trying to setup OCSP and am having issues. I have Let's Encrypt Extension and therefore my certificates change frequently. The Let's Encrypt Extension stores the domain certificates at:

/usr/local/psa/var/modules/letsencrypt/etc/live/yourdomain.tld/​

I added the following under additional nginx directives and am unable to get OCSP stapling working... And of course I replaced mydomain.tld with my actual domain and am able to cat the directory and file to view the pem files.

ssl_certificate /usr/local/psa/var/modules/letsencrypt/etc/live/mydomain.tld/fullchain.pem;
ssl_certificate_key /usr/local/psa/var/modules/letsencrypt/etc/live/mydomain.tld/privkey.pem;
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.4.4 8.8.8.8 valid=300s;
resolver_timeout 10s;​

Thank you very much for your time...

 

[Walter]

Additional info: I also replaced ssl_certificate with ssl_trusted_certificate in the commands above.

To test, I ran the ssl labs test more than once and also submitted the following command on the plesk server and received this response...

echo | openssl s_client -connect mydomain.tld:443 -servername serverfqdn -status

Response:
OCSP response: no response sent​

And of course, the mydomain.tld was replaced with the actual domain and serverfqdn with the fullyqualified plesk server name.

 

[UFHH01]

Hi Walter,

you would certainly use :

    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /usr/local/psa/var/modules/letsencrypt/etc/live/domain.tld/chain.pem;
    resolver 8.8.4.4 8.8.8.8 valid=300s;
    resolver_timeout 10s;

in your additional NGINX directives, which works perfect, as you can see with tests from => Qualys SSL Labs and as well with your command:

echo | openssl s_client -connect mydomain.tld:443 -servername serverfqdn -status

...
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
    Produced At: Apr  4 20:40:00 2017 GMT
...

Pls. make at least TWO tests at "Qualys", because the first one after your changes might not display

OCSP stapling Yes

 

[Walter]

That did not work...

Here is everything in additional nginx directives:

#Enable OCSP
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /usr/local/psa/var/modules/letsencrypt/etc/live/mydomainfolder/chain.pem;
resolver 8.8.4.4 8.8.8.8 valid=300s;
resolver_timeout 10s;

#Necessary to enable php fpm for nginx and wordpress
if (!-e $request_filename) {
    rewrite ^.*$ /index.php last;
}

Clicked apply and then ok and everything appears good. This happened previously. So I went to services management and restarted nginx. The nginx service would stop and not restart. That is where I received the following error:

Unable to start service: Unable to manage service by nginxmng: ('start', 'nginx'). Error: [2017-04-08 11:42:19] ERR [util_exec] proc_close() failed ['/usr/local/psa/admin/bin/nginx_control' '--start'] with exit code [1] Can not start proxy server:

/var/log/nginx/error.log shows the following:

2017/04/08 11:42:17 [emerg] 4638#0: SSL_CTX_load_verify_locations("/usr/local/psa/var/modules/letsencrypt/etc/live/mydomainfolder/chain.pem") failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/usr/local/psa/var/modules/letsencrypt/etc/live/mydomainfolder/chain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib)

# just the line below and then nginx service is able to start

#ssl_trusted_certificate /usr/local/psa/var/modules/letsencrypt/etc/live/mydomainfolder/chain.pem;

To state the obvious, I replaced the actual folder names with mydomainfolder for posting on the internet.

 

[Walter]

ls -ltr /usr/local/psa/var/modules/letsencrypt/etc/live/mydomainfolder
total 4
-rw-r--r--. 1 psaadm psaadm 423 Apr 6 08:59 README
lrwxrwxrwx. 1 psaadm psaadm 45 Apr 6 08:59 privkey.pem -> ../../archive/mydomainfolder/privkey1.pem
lrwxrwxrwx. 1 psaadm psaadm 47 Apr 6 08:59 fullchain.pem -> ../../archive/mydomainfolder/fullchain1.pem
lrwxrwxrwx. 1 psaadm psaadm 43 Apr 6 08:59 chain.pem -> ../../archive/mydomainfolder/chain1.pem
lrwxrwxrwx. 1 psaadm psaadm 42 Apr 6 08:59 cert.pem -> ../../archive/mydomainfolder/cert1.pem

cd /usr/local/psa/var/modules/letsencrypt/etc/archive/mydomainfolder/
ls -ltr
total 16
-rw-r--r--. 1 psaadm psaadm 1708 Apr 6 08:59 privkey1.pem
-rw-r--r--. 1 psaadm psaadm 3634 Apr 6 08:59 fullchain1.pem
-rw-r--r--. 1 psaadm psaadm 1646 Apr 6 08:59 chain1.pem
-rw-r--r--. 1 psaadm psaadm 1987 Apr 6 08:59 cert1.pem

 

[UFHH01]

Hi Walter,

pls. CHECK, if the corresponding "chain.pem" is existent on your server, because your issue:

2017/04/08 11:42:17 [emerg] 4638#0: SSL_CTX_load_verify_locations("/usr/local/psa/var/modules/letsencrypt/etc/live/mydomainfolder/chain.pem") failed (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/usr/local/psa/var/modules/letsencrypt/etc/live/mydomainfolder/chain.pem','r') error:2006D002:BIO routines:BIO_new_file:system lib error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib)

states clear, that this file can't be loaded. => SSL_CTX_load_verify_locations ... failed
Another hint to your issue is, that when you comment out the line, nginx starts without issues.

Pls. , what is the output of the command ( logged in as user "root" over SSH ):

ls -lah /usr/local/psa/var/modules/letsencrypt/etc/live/mydomainfolder

Edit: Well you just added some more informations right before I posted ^^


Edit 2: Consider as well to check the "user" in your nginx - conf and as well the user+group of the current runnung processes:

grep user /etc/nginx/nginx.conf

AND

ps -eo pid,comm,euser,supgrp | grep nginx

 

[Walter]

Using info above, I created /etc/ssl/stapling folder and then copied out the chain.pem file from /usr/local/psa/var/modules/letsencrypt/etc/archive/mydomainfolder/

I ran the command 2x

echo | openssl s_client -connect mydomain.tld:443 -servername serverfqdn -status

And it worked...

OCSP response:
======================================
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3

So we know for sure the certificate itself works and that this must be a permissions issue.

grep user /etc/nginx/nginx.conf
#user  nginx;
    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

ps -eo pid,comm,euser,supgrp | grep nginx
#

 

[Walter]

Permissions of workaround folder:

ls -lah /etc/ssl
total 20K
drwxr-xr-x.   2 root root 4.0K Apr  8 12:27 stapling

Permissions of workaround pem file:

ls -lah /etc/ssl/stapling/chain.pem
-rw-r--r--. 1 root root 1.7K Apr  8 12:27 /etc/ssl/stapling/chain.pem

I copied the file back into the /usr/local/psa/var/modules/letsencrypt/etc/live/mydomainfolder directory as chaintest.pem

ls -ltr
total 8
-rw-r--r--. 1 psaadm psaadm  423 Apr  6 08:59 README
lrwxrwxrwx. 1 psaadm psaadm   45 Apr  6 08:59 privkey.pem -> ../../archive/mydomainfolder/privkey1.pem
lrwxrwxrwx. 1 psaadm psaadm   47 Apr  6 08:59 fullchain.pem -> ../../archive/mydomainfolder/fullchain1.pem
lrwxrwxrwx. 1 psaadm psaadm   43 Apr  6 08:59 chain.pem -> ../../archive/mydomainfolder/chain1.pem
lrwxrwxrwx. 1 psaadm psaadm   42 Apr  6 08:59 cert.pem -> ../../archive/mydomainfolder/cert1.pem
-rw-r--r--. 1 root   root   1646 Apr  8 12:37 chaintest.pem

I updated nginx conf additional directives with:
ssl_trusted_certificate /usr/local/psa/var/modules/letsencrypt/etc/live/mydomainfolder/chaintest.pem;
nginx not starting again.

 

[Walter]

I created a bug report with the letsencrypt plesk extension.

LE 2.0.2 Rel 29 - nginx permission errors trying to reference Let's Encrypt certificates within Plesk Onyx nginx additional directives · Issue #161 · plesk/letsencrypt-plesk · GitHub

There shouldn't be any reason why Plesk/nginx can't read the /usr/local/psa/var/modules/letsencrypt/etc/live/mydomainfolder/ directories. Afterall, the stated purpose in the readme file for chain.pem is to be used for OCSP stapling.

UFHH01, thank you so much for your time. I'm certainly willing to go further with troubleshooting with you so we can post the exact solution that Plesk needs to do to make the LE extension work here. I am though hesitant with testing various permissions on my production instance. I know I could create a symlink to the workaround folder for a solution right now but I'd rather do this right without workaround.

 

[UFHH01]

Hi Walter,

grep user /etc/nginx/nginx.conf
#user nginx;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '"$http_user_agent" "$http_x_forwarded_for"';

Pls. consider to uncomment "#user nginx;", so that nginx will run as user "nginx".

ps -eo pid,comm,euser,supgrp | grep nginx
#

There will certainly ONLY be an output for that command on your command line, if nginx is currently running!

 

[Walter]

I uncommented user nginx; and successfully restarted the nginx service. Went into additional directives and uncommented pem location, clicked apply which seemed to place it in a perpetual submitting state where the "apply" button says "please wait" and received this error in gui:

Internal error ;-P
ERROR: Zend_Controller_Exception: Permission denied.#0 /usr/local/psa/admin/externals/Zend/Controller/Plugin/Broker.php(309): Plesk\Application\Controller\Plugin\ForgeryProtection-&gt;preDispatch(Object(Zend_Controller_Request_Http))<br />
#1 /usr/local/psa/admin/externals/Zend/Controller/Front.php(941): Zend_Controller_Plugin_Broker-&gt;preDispatch(Object(Zend_Controller_Request_Http))<br />
#2 /usr/local/psa/admin/plib/Application/Web.php(39): Zend_Controller_Front-&gt;dispatch(NULL)<br />
#3 /usr/local/psa/admin/htdocs/application.php(15): Plesk\Application_Web-&gt;run()<br />
#4 &#123;main&#125; (Broker.php:312)<br />
<br><a href='http://kb.plesk.com/plesk-error/search?metaId=b2d9cf435e3de4f4af59765e13a75929&messageId=c66c505d2e0449401f733a4b1a551f8b&file=Broker.php&line=312&type=Zend_Controller_Exception&version=17.0.17&message=Permission+denied.%23+%2Fpath%2Fapplication.php%28%29%3A+Plesk%2FApplic' target='_blank'>Search for related Knowledge Base articles</a>

Command you asked me to submit (With nginx running! ;-) ):

ps -eo pid,comm,euser,supgrp | grep nginx
15508 nginx           root     -
15509 nginx           nginx    nginx,psaserv

Commenting out the pem entry in additional directives and restarting nginx gets it back to a working state.