Plesk Certificate Generation = 502 bad gateway

[Sefu]

Hello,

I'm running Plesk 12.5.30 on Ubuntu 12.04.4 LTS‬.

I've been pulling my hair out over this for a couple of days...

I'd like to update my (expired) SSL certificates, but the thing is, even when trying to create an 'experimental' one, every time I hit 'self-signed' I get a "502 Bad Gateway" message, with 'nginx' underneath it... the thing is, I don't even have nginx running!

I've already searched all existing processes and logs for instances of nginx and that error, but nothing is turning up.

Can someone more enlightened opine or this, or at least tell me where to look?

Thanks, and best regards.

Update: I found this in my /var/log/sw-cp-server/error_log:

2016/03/09 10:18:05 [error] 26873#0: *196 upstream sent unexpected FastCGI record: 3 while reading response header from upstream, client: *.*.*.*, server: , request: "POST /smb/ssl-certificate/add/id/2 HTTP/1.1", upstream: "fastcgi://unix:/var/run/sw-engine.sock:", host: "my-domain.com:8443", referrer: "https://my-domain.com:8443/smb/ssl-certificate/add/id/2"

 

[Sefu]

I think I found it.

[10-Mar-2016 11:35:36 Europe/Berlin] PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 4294967296 bytes) in Unknown on line 0

I understand that Plesk runs in its own apache environment, which would mean that it would have its own php.ini file as well?

And what the (expletive) would try to allocate 4gb of memory?!

So...

I put it up to several settings (8192M, 4096M, 5120M)

Out of memory (allocated 262144) (tried to allocate 4294967296 bytes) in Unknown on line 0

It's trying to use 4g of memory, and if the setting is smaller it doesn't even try, and when the setting is above the limit and it does try, it finds that the existing (physical) memory exceeds that needed.

The problem is Plesk trying to allocate 4g - that's just weird and uncalled for.

Plodding on... this may be related to mysql trying to call/create the 'longtext' required for the completed cipher, and in php5.3 installations (the server is using 5.3.10), it tries to allocate the 'longtext' 4g max limit so it won't 'lose anything'. I may have to update my version of php ; P

This is a known limitation of ext/mysqli when using libmysql (always in 5.2 and previous) and when libmysql is enabled with 5.3 . The reason is that the server sends not too specific metadata about the column. This longtext has a max length of 4G and ext/mysqli tries to bind with the max length, to be sure no data loss occurs (data doesn't fit in the bind buffer on C level). However, that means 4G for a longtext/longblob column. ext/mysqli has been changed to have a way to work around that. You need to call mysqli_stmt_store_result() which will store the data locally, which means, of course a higher memory usage for PHP. However, because you use libmysql this won't hit the PHP's memory limit, for sure. During store_result the max_length of every column will be calculated and then when bind_result is executed only a buffer with size of max_length will be allocated, which will be definitely lower than 4G. In short, prepare execute store_result bind_result fetch...fetch...fetch

That rings a -lot- of bells, and it would make sense that the update could do that, but it could be something else. Still looking....

...that seems to be it. I've installed the newer php versions, now to get Plesk panel to switch over to them (no documentation on this!).
Thanks for your input - this was a 'particular' problem (Plesk 12.5 on Php 5.3), but I hope this will be useful to anyone else updating Plesk on older servers/installations.

 

[Sefu]

I'm back at square one, because mysql is not the cause of the problem (nothing in the logs).

Yet something is requesting 4g of memory (that my server doesn't have) - can anyone advise me on how to troubleshoot this?

Best, and thanks.

 

[Sefu]

Is there perhaps any way of re-routing Plesk's nginx requests to swap instead of memory (ram)?

Another question: I wasn't able to find a mysql error, but if plesk is 'setting up' a 'longtext' write (and allocating the full 'required' 4go memory for that task), would it even generate one?

I would have to look into the mechanics of how Plesk generates certificates to better understand that.

 

[Lloyd_mcse]

Hi Sefu,
have you tried this...

https://kb.plesk.com/en/118757

Just to clarify, Plesk uses it's own web server based on nginx, you can find the config files in ...

Plesk Panel Web Server - /etc/sw-cp-server/

Plesk Panel FPM - /etc/sw-engine/

See the staus of them with...

# service sw-cp-server status
# service sw-engine status

Logs are located in...

/var/log/sw-cp-server/

And domains are served via the standard OS Packages in...

/etc/apache2/
/etc/nginx/

Let me know if the above kb helps.
Regards

Lloyd

 

[Sefu]

Thank you so much.

Yes, I have set the buffer, etc up:

fastcgi_buffers 32 32k;
fastcgi_buffer_size 64k;

...and yes, in the right one (/etc/sw-cp-server/config ; )

:~# service sw-cp-server status
* sw-cp-serverd is running

:~# service sw-engine status
* sw-engine-fpm is running

...and both services are running.

Upon certificate creation attempt:

The sw-engine.log:

[11-Mar-2016 14:29:46] WARNING: [pool plesk] child 30565 exited with code 70 after 54.602890 seconds from start
[11-Mar-2016 14:29:46] NOTICE: [pool plesk] child 30768 started

Same instance, from the sw-cp-server error log:

2016/03/11 14:29:46 [error] 28410#0: *238 upstream sent unexpected FastCGI record: 3 while reading response header from upstream, client: **.***.***.**, server: , request: "POST /admin/ssl-certificate/add HTTP/1.1", upstream: "fastcgi://unix:/var/run/sw-engine.sock:", host: "www.my-server.com:8443", referrer: "https://www.my-server.com:8443/admin/ssl-certificate/add"

Oh, and from the Plesk panel log:

[11-Mar-2016 14:29:46 Europe/Berlin] PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 4294967296 bytes) in Unknown on line 0

 

[trialotto]

@Sefu,

Are you using Let´s Encrypt extension? Or do you have some other request to a certificate provider?

Please provide some relevant output from the logs.

Regards

 

[Sefu]

Thanks for your reply.

Are you using Let´s Encrypt extension? Or do you have some other request to a certificate provider?

No, and no: all this is happening for a self-signed certificate.

 

[trialotto]

@Sefu,

The essence of your problem is that something is causing a "hickup" in your FastCGI process.

The problem is only related to the Plesk Panel, so we can focus on that.

The problem can be the result from a PHP bug, a simple server setting, a 4096 bit certificate, lack of resources on the machine and so on.

Please do the following:

1) run the commands:

- service psa stop
- service psa start

and have a look whether the issue persists.

2) provide the output of the command: free -hm

3) check that openssl is not running: ps aux | grep openssl (that is, we want to make sure that the certificate generation process is not still running)

4) most important: check that the fastcgi script is in place, there should be a file /opt/psa/admin/htdocs/application.php

5) check that Plesk Panel´s configuration is pointing to that fastcgi script: have a look at the contents of /etc/sw-cp-server/conf.d/plesk.inc


In general, I would recommend to

- follow the steps above, to exclude some common issues (if they are present, a regeneration of SSL certificate will also cause problems)
- create a new self-signed certificate (2048 bit) and apply that (and it is very likely that the new certificate will not cause any issues)
- remove the old self-signed certificate (afterwards)

Anyway, just let me know.

Regards...