Resolved SSL: error:0906D06C:PEM routines:PEM_read_bio:no start line

[Dukemaster]

Hi,
like every sunday a cronjob creating dhparam certificates was successfully done, but the all 4 .pem(s) in /etc/dhparam are empty with 0 byte.
nginx -t failed

root@server:~# nginx -t
nginx: [emerg] PEM_read_bio_DHparams("/etc/dhparam/dhparam4096.pem") failed (SSL: error:0906D06C:PEM routines:PEM_read_bio:no start line:Expecting: DH PARAMETERS)
nginx: configuration file /etc/nginx/nginx.conf test failed

here another

systemctl status nginx.service
● nginx.service - Startup script for nginx service
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2017-10-08 22:30:35 CEST; 42min ago
  Process: 7237 ExecStop=/bin/kill -s QUIT $MAINPID (code=exited, status=0/SUCCESS)
  Process: 2200 ExecReload=/bin/kill -s HUP $MAINPID (code=exited, status=0/SUCCESS)
  Process: 2196 ExecReload=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
  Process: 2193 ExecReload=/usr/bin/test $NGINX_ENABLED = yes (code=exited, status=0/SUCCESS)
  Process: 3480 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)
  Process: 7246 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=1/FAILURE)
  Process: 7242 ExecStartPre=/usr/bin/test $NGINX_ENABLED = yes (code=exited, status=0/SUCCESS)
Main PID: 3484 (code=exited, status=0/SUCCESS)

Oct 08 22:30:35 server.example.com systemd[1]: Starting Startup script for nginx service...
Oct 08 22:30:35 server.example.com nginx[7246]: nginx: [emerg] PEM_read_bio_DHparams("/etc/dhparam/dhparam4096.pem") failed (SSL: error:0906D06C:PEM routines:PEM_read_bio:no start line:Expectin
Oct 08 22:30:35 server.example.com nginx[7246]: nginx: configuration file /etc/nginx/nginx.conf test failed
Oct 08 22:30:35 server.example.com systemd[1]: nginx.service: Control process exited, code=exited status=1
Oct 08 22:30:35 server.example.com systemd[1]: Failed to start Startup script for nginx service.
Oct 08 22:30:35 server.example.com systemd[1]: nginx.service: Unit entered failed state.
Oct 08 22:30:35 server.example.com systemd[1]: nginx.service: Failed with result 'exit-code'.

Do you know how to fix the error or restore the configuration.

Lots of greets

 

[IgorG]

Please make sure that you have the same crontask:

# cat /etc/cron.weekly/gen_dhparam
#!/bin/bash

mkdir -p /etc/dhparam 2>/dev/null
FILE=`mktemp`

N=512
while [ $N -le 4096 ] ; do
  openssl dhparam $N -out $FILE && cat $FILE >/etc/dhparam/dhparam${N}.pem
  let N*=2
done

rm -f ${FILE}

As result:

# ll /etc/dhparam
total 16
-rw-r--r-- 1 root root 245 Oct 8 04:22 dhparam1024.pem
-rw-r--r-- 1 root root 424 Oct 8 04:22 dhparam2048.pem
-rw-r--r-- 1 root root 769 Oct 8 04:29 dhparam4096.pem
-rw-r--r-- 1 root root 156 Oct 8 04:22 dhparam512.pem

 

[Dukemaster]

Thanks a lot @IgorG
No I haven't. Think I have another cron job/task. Something is wrong.
Could You please help me create it?

root@server:~# cat /etc/cron.weekly/gen_dhparam
cat: /etc/cron.weekly/gen_dhparam: No such file or directory

This is my cron task in cron tab shell via Plesk panel as /bin/bash (chrooted), time sunday 4.00:
Yesterday I started this cron task manually and it was successfully but seems with no result.

FILE=`mktemp` ; openssl dhparam 512 -out $FILE && mv -f $FILE /etc/dhparam/dhparam512.pem && FILE=`mktemp` ; openssl dhparam 1024 -out $FILE && mv -f $FILE /etc/dhparam/dhparam1024.pem && FILE=`mktemp` ; openssl dhparam 2048 -out $FILE && mv -f $FILE /etc/dhparam/dhparam2048.pem && FILE=`mktemp` ; openssl dhparam 4096 -out $FILE && mv -f $FILE /etc/dhparam/dhparam4096.pem

Example from yesterday:

Generating DH parameters, 512 bit long safe prime, generator 2
This is going to take a long time
.............+......................................+..................................................++*++*++*++*++*++*
Generating DH parameters, 1024 bit long safe prime, generator 2
This is going to take a long time
.............+...................................................................+...................+.............................................+..+.............................+...+......................................................................+......................................................+.......................................

Greets

 

[IgorG]

Just create this file with this content and run it. Files should be created.

 

[Dukemaster]

I created this file, changed permissions to 755 and ran it by "~# sh /etc/cron.weekly/gen_dhparam"
An endless loop of creating again and again started.

/etc/cron.weekly/gen_dhparam: 9: /etc/cron.weekly/gen_dhparam: let: not found
Generating DH parameters, 512 bit long safe prime, generator 2
This is going to take a long time
.......+.........+......+...........................+...................+...................................................+........................................................+.........+................+.+.........+....+......................................................................................................................................................................+.................+...........+..........+....................................+................................+.......+....................................................+..............................................+............+..............................................+......................+................................+........................................+.........+..................................................+.................................+........+..+.....................................................+....................+..........+................+.......+.......................................+.....................+....................++*++*++*++*++*++*
/etc/cron.weekly/gen_dhparam: 9: /etc/cron.weekly/gen_dhparam: let: not found
Generating DH parameters, 512 bit long safe prime, generator 2
This is going to take a long time
...............+......+...+...+........+...............+..................................................+..........+....................................+.+..............................+....................+.................................................................+..........................+...........................+................+.............+........+...+..................+......................................................+..............+................................................+.+.........................+..+.........+.......+.....................................+...............+.+...+....+...............................+.......+............+...+............................................................................................+............+..............................+..+........................................................................................................+..................................................+..................+..+...............................++*++*++*++*++*++*
/etc/cron.weekly/gen_dhparam: 9: /etc/cron.weekly/gen_dhparam: let: not found
Generating DH parameters, 512 bit long safe prime, generator 2
This is going to take a long time....and so on

Interesting result was, as I stopped the script by STRG C the dhparam512.pem had content

-----BEGIN DH PARAMETERS-----
MEYkkkkkkkkkkkkkkkkkkk--- whole content changed by me for security reasons ---- kkkkkkkkkkkkkkkkkkkkk+4k/S7ECAe
-----END DH PARAMETERS-----

Perhaps the error message is not correct, or I should run it for the usual cron tab time which is little less than one hour for my weekly cron task in Plesk panel, or perhaps something else is wrong.
Now I started it over Plesk panel manually:

Führe Aufgabe "FILE=`mktemp` ; openssl dhparam 512 -out $FILE && mv -f $FILE /etc/dhparam/dhparam512.pem && FILE=`mktemp` ; openssl dhparam 1024 -out $FILE && mv -f $FILE /etc/dhparam/dhparam1024.pem && FILE=`mktemp` ; openssl dhparam 2048 -out $FILE && mv -f $FILE /etc/dhparam/dhparam2048.pem && FILE=`mktemp` ; openssl dhparam 4096 -out $FILE && mv -f $FILE /etc/dhparam/dhparam4096.pem" aus...

Don't know if it will be successfully done or not.
Thanks for help and Greets.

 

[Dukemaster]

Now I know the reason why.
The (old) scheduled task is removing whole content (certificates) of all 4 .pem files in /etc/dhparam (dhparam512.pem, dhparam1024.pem, dhparam2048.pem and dhparam4096.pem).
But how to create all of them?
Your script @IgorG is creating only certificate for dhparam512.pem, not for the important others.
Would be great if You or someone could help me further. Could be related to the new openssl versions I installed last week.
Greets

 

[Dukemaster]

Dirtiest of dirty fixes:
As I recognized yesterday that something is going wrong I was able to backup the old certificates before they were removed by the scheduled task.
Now, I only copy and paste the certs into the depending .pem files, I was able to test nginx and restart service nginx like usual.
Okay, for now websites are running. I also reinstalled Plesk with patches, which wasn't necessary but it was good for my nerves.
But for security reasons it would be important at all to fix the problem why the certificates can't be created by the cron task / scheduled task which worked for months.
Everything points to perhaps a little misconfiguration or wrong permission in openssl or nginx.
dpkg -l and aptitude say both everything is well installed and running. Thanks for help to @IgorG and @UFHH01 for the help in openssl, nginx and more.
Does someone knows how to find out my mistake or a bug as a second possibility?

 

[UFHH01]

Hi Dukemaster,

the previous provided suggestion from => #2 for example, works still as expected and there is no need to change the corresponding crontab.

However, issues and problems are always possible and you might experience once in a while, that for some reason, a crontab didn't work, or didn't finish as expected. I think this is no big deal at all and you should rather consider to run the command again ( manually over the command line or/and over your Plesk Control Panel ), in order to see, if the issues appears a second time! Only if you are able to reproduce an issue/error/problem, you would continue to investigate a possible root cause.

 

[Dukemaster]

Hi @UFHH01
No, my crontab is like months ago. I didn't changed it nor deleted it. But I it starts working by manually start and a loop is started because of (see above), yes they appeared twice or three times today.
Crontab deletes the content of all 4 and creates only the 512.pem content.
/etc/cron.weekly/gen_dhparam: 9: /etc/cron.weekly/gen_dhparam: let: not found

 

[UFHH01]

Hi Dukemaster,

you are mixing issues/errors/problems again, Dukemaster.

/etc/cron.weekly/gen_dhparam: 9: /etc/cron.weekly/gen_dhparam: let: not found

"let" is a bash builtin command. By default, on Debian/Ubuntu based systems, your shell is actually a "dash" shell, which you are able to reconfigure with the command ( logged in as user "root" over SSH ):

dpkg-reconfigure dash

The system shell is the default command interpreter for shell scripts.

Use dash as the default system shell (/bin/sh)?

Preferred/recommended answer:

<No>

 

[IgorG]

@Dukemaster please read my answer to your question in PM. There is full instruction for you

 

[Dukemaster]

Hi @IgorG , thanks for Your help.
I followed your advice and created file /usr/local/sbin/gen_dhparam (with # chmod +x).
Ran it by # /usr/local/sbin/gen_dhparam
and all 4 .pem(s) were created/filled with new certificates.
Then nginx -t and service nginx restart worked.
Also the same gen_param I created in /etc/cron.weekly with same permissions.
Don't know if it is the symlink You told me to create.
But I have TLS 1.3 with draft 18 again in Qualys and Firefox says also TLS 1.3 is used.
THANKS A LOT
Would be great if you could tell me if the symbolic link is what You suggested.

My results are a little different from yours:
~# ll /etc/dhparam
total 32
drwxr-xr-x 2 root root 4096 Oct 9 15:38 ./
drwxr-xr-x 127 root root 12288 Oct 10 21:05 ../
-rw-r--r-- 1 root root 245 Oct 10 11:16 dhparam1024.pem
-rw-r--r-- 1 root root 424 Oct 10 11:17 dhparam2048.pem
-rw-r--r-- 1 root root 769 Oct 10 11:58 dhparam4096.pem
-rw-r--r-- 1 root root 156 Oct 10 11:16 dhparam512.pem

~# ll /etc/cron.weekly/gen_dhparam
-rwxr-xr-x 1 root root 200 Oct 9 13:52 /etc/cron.weekly/gen_dhparam*

THANKS ALSO to @UFHH01. You were hard again to me , but in most deeper meanings I agree with that what you want me to tell.
I also informed myself about dash and system shell, that in special cases like installing docker (not on Plesk systems) it is necessary to switch from dash to bash by reconfiguring dash. But You don't suggested me to reconfigure dash, or am I wrong, as a question?

 

[IgorG]

Would be great if you could tell me if the symbolic link is what You suggested.

Just create symlink with

# ln -s /usr/local/sbin/gen_dhparam /etc/cron.weekly/

 

[UFHH01]

Hi Dukemaster,

But You don't suggested me to reconfigure dash, or am I wrong, as a question?

Why not? If you desire more informations about "dash" and "bash", pls. consider to use a Google search, as there are decent articles out there. In addition, you will notice, that all your Plesk - related accounts are configured to use "bash" and not "dash".

 

[Dukemaster]

Thanks for Your amazing help both of You @IgorG and @UFHH01.
My symlink was wrong, now I corrected also this.

 

[Dukemaster]

which you are able to reconfigure with the command ( logged in as user "root" over SSH ):

dpkg-reconfigure dash

Thanks again @UFHH01 I followed your recommendation:

root@server:~# dpkg-reconfigure dash
Removing 'diversion of /bin/sh to /bin/sh.distrib by dash'
Adding 'diversion of /bin/sh to /bin/sh.distrib by bash'
Removing 'diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash'
Adding 'diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by bash'

It does what You expected.
Greets

 

[Dukemaster]

Thanks a lot @IgorG and @UFHH01
Yesterday mail from system about successful weekly cron task in PLESK PANEL:

/etc/cron.weekly/gen_dhparam:
Generating DH parameters, 512 bit long safe prime, generator 2
This is going to take a long time
...............................+.....................................+.........+...........+...+..........................more...
Generating DH parameters, 1024 bit long safe prime, generator 2
This is going to take a long time
..........................................................+......................................................+.......................more....
Generating DH parameters, 2048 bit long safe prime, generator 2
This is going to take a long time
..........................................+.......................................................................................+.............+....+...........+.....more
Generating DH parameters, 4096 bit long safe prime, generator 2
This is going to take a long time
................................................more.....................

It works again...