nginx 502 error

[omexlu12]

Hello @IgorG , @Lloyd_mcse

Every morning arround 6am i get a nginx 502 on all websites hosted by plesk and to solve the problem i must to this (panel dont have any 502 error):
http://kb.odin.com/de/123735

After that all works good until the next morning.
The nginx logs are empty but i remarked this error in proxy_error_log:

2015/11/12 09:17:47 [error] 31126#0: *13 connect() failed (111: Connection refused) while connecting to upstream, client: 35.0.127.52, server: domain.tld, request: "GET /favicon.ico HTTP/1.1", upstream: "https://IP:7081/favicon.ico", host: "domain.tld"

Is is maybe a solution to write this into nginx directives from every webspace:
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;

What can i do to fix this problem on debian jessie permently?

EDIT the only thing i have changed is the "nginxDomainVirtualHost.php" (custom):

<?php
/**
* @var Template_VariableAccessor $VAR
* @var array $OPT
*/
?>
server {
    listen <?php echo $OPT['ipAddress']->escapedAddress . ':' . $OPT['frontendPort'] .
        ($OPT['default'] ? ' default_server' : '') . ($OPT['ssl'] ? ' ssl' : '') ?>;

    server_name <?php echo $VAR->domain->asciiName ?>;
<?php if ($VAR->domain->isWildcard): ?>
    server_name ~^<?php echo $VAR->domain->pcreName ?>$;
<?php else: ?>
    server_name www.<?php echo $VAR->domain->asciiName ?>;
<?php   if ($OPT['ipAddress']->isIpV6()): ?>
    server_name ipv6.<?php echo $VAR->domain->asciiName ?>;
<?php   else: ?>
    server_name ipv4.<?php echo $VAR->domain->asciiName ?>;
<?php   endif ?>
<?php endif ?>
<?php if ($VAR->domain->webAliases): ?>
<?php   foreach ($VAR->domain->webAliases as $alias): ?>
    server_name <?php echo $alias->asciiName ?>;
    server_name www.<?php echo $alias->asciiName ?>;
<?php   endforeach ?>
<?php endif ?>
<?php if ($VAR->domain->previewDomainName): ?>
    server_name "<?php echo $VAR->domain->previewDomainName ?>";
<?php endif ?>

<?php if ($OPT['ssl']): ?>
<?php $sslCertificate = $VAR->server->sni && $VAR->domain->physicalHosting->sslCertificate ?
    $VAR->domain->physicalHosting->sslCertificate :
    $OPT['ipAddress']->sslCertificate; ?>
<?php   if ($sslCertificate->ce): ?>
    ssl_certificate             <?php echo $sslCertificate->ceFilePath ?>;
    ssl_certificate_key         <?php echo $sslCertificate->ceFilePath ?>;
    ssl_session_cache                  shared:SSL:5m;
    ssl_dhparam    /etc/ssl/RSA2048.pem;
<?php       if ($sslCertificate->ca): ?>
    ssl_client_certificate      <?php echo $sslCertificate->caFilePath ?>;
<?php       endif ?>
<?php   endif ?>
<?php endif ?>

<?php if (!empty($VAR->domain->physicalHosting->proxySettings['nginxClientMaxBodySize'])): ?>
    client_max_body_size <?php echo $VAR->domain->physicalHosting->proxySettings['nginxClientMaxBodySize'] ?>;
<?php endif; ?>

<?php if ($VAR->domain->physicalHosting->scriptTimeout): ?>
    proxy_read_timeout <?php echo min($VAR->domain->physicalHosting->scriptTimeout, 2147483); ?>;
<?php endif; ?>

    root "<?php echo $OPT['ssl'] ? $VAR->domain->physicalHosting->httpsDir : $VAR->domain->physicalHosting->httpDir ?>";
    access_log "<?php echo $VAR->domain->physicalHosting->logsDir . '/' . ($OPT['ssl'] ? 'proxy_access_ssl_log' : 'proxy_access_log') ?>";
    error_log "<?php echo $VAR->domain->physicalHosting->logsDir . '/proxy_error_log' ?>";

<?php if ($OPT['default']): ?>
<?php echo $VAR->includeTemplate('service/nginxSitePreview.php') ?>
<?php endif ?>

<?php echo $VAR->domain->physicalHosting->proxySettings['allowDeny'] ?>

<?php echo $VAR->includeTemplate('domain/service/nginxSeoSafeRedirects.php', array('ssl' => $OPT['ssl'])); ?>

    location / {
<?php echo $VAR->includeTemplate('domain/service/proxy.php', $OPT); ?>
    }

<?php if (!$VAR->domain->physicalHosting->proxySettings['nginxTransparentMode'] && !$VAR->domain->physicalHosting->proxySettings['nginxServeStatic']): ?>
    location /internal-nginx-static-location/ {
        alias <?php echo $OPT['documentRoot'] ?>/;
        internal;
    }
<?php endif ?>

<?php if ($VAR->domain->active && !$VAR->domain->physicalHosting->proxySettings['nginxTransparentMode']): ?>

<?php if ($VAR->domain->physicalHosting->php && $VAR->domain->physicalHosting->proxySettings['nginxServePhp']
            || $VAR->domain->physicalHosting->proxySettings['nginxServeStatic']): ?>

<?php if ($VAR->domain->physicalHosting->proxySettings['fileSharingPrefix']): ?>
    location ~ ^/<?php echo $VAR->domain->physicalHosting->proxySettings['fileSharingPrefix'] ?>/ {
<?php echo $VAR->includeTemplate('domain/service/proxy.php', $OPT); ?>
    }
<?php endif; ?>

<?php endif; ?>

<?php if ($VAR->domain->physicalHosting->proxySettings['nginxServeStatic']): ?>

    location @fallback {
<?php echo $VAR->includeTemplate('domain/service/proxy.php', $OPT); ?>
    }

<?php echo $VAR->includeTemplate('domain/service/nginxProtectedDirectories.php', $OPT); ?>

    location ~ ^/(.*\.(<?php echo $VAR->domain->physicalHosting->proxySettings['nginxStaticExtensions'] ?>))$ {
        try_files $uri @fallback;
    }
<?php endif ?>

<?php if ($VAR->domain->physicalHosting->php && $VAR->domain->physicalHosting->proxySettings['nginxServePhp']): ?>

<?php if ($VAR->domain->physicalHosting->hasWebstat): ?>
    location ~ ^/(plesk-stat|webstat|webstat-ssl|ftpstat|anon_ftpstat|awstats-icon) {
        <?php echo $VAR->includeTemplate('domain/service/proxy.php', $OPT); ?>
    }
<?php endif; ?>

    location ~ ^/~(.+?)(/.*?\.php)(/.*)?$ {
        alias <?php echo $VAR->domain->physicalHosting->webUsersDir ?>/$1/$2;
        <?php echo $VAR->includeTemplate('domain/service/fpm.php'); ?>
    }

    location ~ ^/~(.+?)(/.*)?$ {
        <?php echo $VAR->includeTemplate('domain/service/proxy.php', $OPT); ?>
    }

    <?php echo $VAR->includeTemplate('domain/service/nginxWordpress.php', $OPT); ?>

    location ~ \.php(/.*)?$ {
        <?php echo $VAR->includeTemplate('domain/service/fpm.php'); ?>
    }

    <?php echo $VAR->includeTemplate('domain/service/nginxWordpressIndexing.php', $OPT); ?>

    location ~ /$ {
        <?php echo $VAR->domain->physicalHosting->proxySettings['directoryIndex'] ?>
    }
<?php endif ?>

<?php endif ?>

<?php if (is_file($VAR->domain->physicalHosting->customNginxConfigFile)): ?>
    include "<?php echo $VAR->domain->physicalHosting->customNginxConfigFile ?>";
<?php endif; ?>
}

Adding this into it:

<?php if ($sslCertificate->ce): ?>
ssl_certificate <?php echo $sslCertificate->ceFilePath ?>;
ssl_certificate_key <?php echo $sslCertificate->ceFilePath ?>;
ssl_session_cache shared:SSL:5m;
ssl_dhparam /etc/ssl/RSA2048.pem;

Maybe the 2 rows in bold are not on the correct place that it makes an error?

Thanks

 

[ehrenwert]

Same error here. Would be great to get a solution for that.

 

[omexlu12]

I can fix this problem with this on the nginx addition settings in the panel on every domain/webspace:
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;

 

[trialotto]

@Omelux12 (and @ehrenwert),

You have added to the nginx config template:

ssl_session_cache shared:SSL:5m;
ssl_dhparam /etc/ssl/RSA2048.pem;

In your case(s), the 502 error is essentially a time-out error, caused by both bad configuration and a time-out configuration issue.

The increase of buffer size is a potential solution to reduce the 502 error, but this will not resolve the issues due to "bad" configuration: Nginx still inefficiently uses resources.

In essence, the line "ssl_dhparam ..;" is not strictly necessary, given the

ssl_certificate <?php echo $sslCertificate->ceFilePath ?>;
ssl_certificate_key <?php echo $sslCertificate->ceFilePath ?>;

directives and, therefore, the line "ssl_dhparam ..;" should be discarded and, at best (read: if desired), replaced by the line "ssl_session_timeout ..;".

However, the "ssl_session_timeout ..;" directive is not actually a solution to the occurring 502 error: it is simply a safeguard.

I am pretty sure that the

ssl_session_cache shared:SSL:5m;

directive does not compare to the settings of the proxy_*_timeout directives in the standard nginx template and/or adjusted (custom) settings.

In short, the part "5m" AND long execution times of processes can cause conflicts, given the proxy_*_timeout directives (or fastcgi_*_timeout directives).

In general, the following has to be mentioned with respect to the hardening of SSL/TLS configuration of Nginx.

Note that one can use the Ephemeral Diffie-Hellman (DHE) key exchange process by implementing the ssl_dhparam directive, but this can be "too much", in the sense that it can or will conflict with the standard ssl related nginx directives (and other nginx config settings) and/or will (heavily) impact performance of the Nginx server.

In specific cases, depending on the role of Nginx server, it is not necessary: nginx as a gateway and/or proxy does not really require a DHE key exchange, whilst the DHE key exchange in nginx running with the php-fpm daemon will result in a performance penalty.

In most cases, a normal form of the Diffie-Hellman (DH) key exchange process can be used (instead of the ephemeral form of key exchange).

Also note that problems with configuration can exist:

- the cipher suite has to be elaborate AND will become very extensive when allowing for backwards compatibility,
- some versions of OpenSSL do not render all the necessary algorithms,
- certificates and keys have to be setup perfectly: nginx relies on OpenSSL and the "danger" of the default 1024-bit OpenSSL key for key exchange exists, in the sense that security of a 2048-bit certificate can be comprised, if the default 1024-bit key is used by mistake,

and so on.

In short, if you really want to implement the ssl_dhparam directive, generate a stronger DHE parameter that is in line with the certificate being used.

Hope the above helps in hardening and/or configuring SSL/TLS for Nginx!

Kind regards......

 

[omexlu12]

Hi @trialotto ,

Yes you are right , the problem still exists but i have difficult to understand your post, what exactly must i do now to solve this 520 Bad Gateway issue?

 

[trialotto]

@omexlu12,

You have two options:

a) remove the lines

ssl_session_cache shared:SSL:5m;
ssl_dhparam /etc/ssl/RSA2048.pem;

from your custom template and/or any custom config.

That way, you are using a standard (i.e. non-ephemeral) key exchange and the dangers of session timeouts are not really present.

OR

b) remove the lines

ssl_certificate <?php echo $sslCertificate->ceFilePath ?>;
ssl_certificate_key <?php echo $sslCertificate->ceFilePath ?>;

from your custom template and/or any config AND

1 - rename the file /etc/nginx/conf.d/ssl.conf (present on at least Plesk 12.5.30) to ssl.conf.bak (or something similar)
2 - create a file ssl-custom.conf in the directory /etc/nginx/conf.d and add the lines:

ssl_ciphers '....<some elaborate set of ciphers, that can or have to be aligned with the intended security targets for Nginx>......';
ssl_prefer_server_ciphers on;

ssl_dhparam <path to appropriate .pem file>

3 - run: nginx -t && service nginx reload


To be honest, option b is very likely to cause many problems:

- for instance, if config files are not adjusted properly and still contain the ssl related directives), AND
- a (heavy) impact on performance is still present: it requires that you setup particular values for the session timeout variables and/or buffer variables, which values will vary across each server used, across each Nginx setup for a specific purpose (a pure proxy setup requires different settings than the setup for nginx with fpm) and so on AND these values have to be adjusted from time to time, in order to minimize the performance decrease.

In conclusion, go for option a (and keep it simple).

Also note that option a is perfectly "acceptable" if you are using Nginx only as a pure proxy.

By the way, one can harden up Nginx security in a more suited method, by implementing (amongst others)

- Http Strict Transport Security (HSTS: enforcing https communication),
- Http Public Key Pinning (Public Key Pinning: assurance that only whitelisted Certificate Authorities can sign certificates for the domains)
- OCSP Stapling (more advanced)

and naturally, the best start to make Nginx secure is removing self-signed certificates and using a proper one from a trusted and good Certificate Authority.

Hope the above helps!

Regards......

 

[omexlu12]

Hi @trialotto ,

Thanks for your detailed answer, i will take a look the next days if the problem persists and then i will give a try with solution A.
This is the standart configuration (template) from plesk so i think there is no problem anymore.

But without this lines i get a bader ranking in SSL test
ssl_session_cache shared:SSL:5m;
ssl_dhparam /etc/ssl/RSA2048.pem;

I will test it the next days and let you know if the problem goes on with this.

PS. HSTS i have activated

 

[omexlu12]

Hi @trialotto ,

I have now remove this line from "/usr/local/psa/admin/conf/templates/custom/domain/nginxDomainVirtualHost.php":
ssl_session_cache shared:SSL:5m;
ssl_dhparam /etc/ssl/RSA2048.pem;

And then:
/usr/local/psa/admin/bin/httpdmng --reconfigure-all

But now my ssl-rating is falling from "A+" to "B":


I will now check if that solve the issue with the 502 - Bad Gateway, if not i will replace the 2 lines again.

What i find strange is that the issue come only at 6:30am, is there any cronjob at this time that makes this issue?

Thanks in advance.

 

[trialotto]

@omexlu12

Note that the execution of

And then: /usr/local/psa/admin/bin/httpdmng --reconfigure-all

has to be followed by the execution of the command "nginx -t && service nginx reload". That is, to be safe (i.e. that Nginx actually does reload all new settings).

As could have been expected, the B grade is (only) the result of DH key exchange implementation, instead of DHE key exchange (as configured with the ssl_dhparam directive).

Anyway, this new (lower) grade does not actually make you more (or less) vulnerable to specific attacks: in essence, security is still sufficient, but it is not to the highest degree (i.e. the highest degree can only be obtained with DHE key exchange, with lots of disadvantages of such an implementation).

In short, no worries, mate.

By the way, compare it with PageSpeed scores: a very high score does not say anything about load times. For instance, it is possible to have a PageSpeed score of 10 to 20%, whilst having a load time of 3ms (thanks to Redis and FastCGI).

In terms of security, the same applies: a higher "security score", as result of DHE key exchange implementation, does make you more vulnerable to distributed attacks, given the fact that DHE is associated with a heavy performance penalty. Certainly when Nginx (only) functions as a "security shield", a vulnerability to potential overloads due to distributed attacks is not really optimal in terms of security.

Really, I love the idea and attempts to make Nginx "NSA proof" (a little bit of an overstatement), but be aware of the facts that (on the one hand) normal undesirable traffic often comes in a simple form, like spam mail and/or simple brute forcing (on standard ports) and that (on the other hand) sophisticated hackers tend to aim at the most secure servers, with the idea that something valuable can be obtained, if something is hidden behind a lot of security measures (i.e. it is some kind of a challenge and sport).

Regards.....

 

[omexlu12]

Hi @trialotto ,

Thanks for your explication, i will look now if the issue is gone with this modification tomorrow and i let you know .
I hope its gone with that and all is good then for me

Thanks

 

[omexlu12]

Hello again @trialotto ,

As expected i have again this error at 6:32 AM (same time) at all websites:


This error has nothing to do with this lines:
ssl_session_cache shared:SSL:5m;
ssl_dhparam /etc/ssl/RSA2048.pem;

Is the Plesk Auto-Update check running at this time, maybe this is the problem?
I dont really know what i can do, else? Really anoying this problem

I consider to turn of reverse proxy and only use apache as webserver

Thanks in advance.

 

[IgorG]

Why you can't check your system crontasks for this specific time?
Do you have something suspicious there?

 

[omexlu12]

Hi @IgorG

I have this in crontabs:

17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

/etc/cron.daily# ls
50plesk-daily 60sa-update apache2 apt aptitude bsdmainutils dpkg drweb-update logrotate man-db mlocate passwd spamassassin webalizer

Or what do you mean exactly?
Are there any logs for the crons?

The Problem is very strange i am helples.

EDIT: All on server is default. I dont understand why i get this error with nginx on websites at this time

 

[IgorG]

25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )

This task is started at 6.25. So in this cron.daily a lot of hardloaded tasks like logrotate, statisctics calculation, etc. This can lead to lack of resources in this time, for example. Is this Plesk installed on VPS or on standalone server?

 

[omexlu12]

@IgorG

Yes there are many cron at daily but the problem is only with the nginx 502 error , maybe i consider to deactivate nginx as reverse proxy.
Plesk is installed on a VPS:

• Intel® Xeon® Intel® Xeon® E5-2660V3
• 12 GB DDR4 ECC RAM
• KVM-Technologie

https://www.netcup.de/bestellen/produkt.php?produkt=1103

But without this steps the websites dont run again:
http://kb.odin.com/de/123735

EDIT: But i think is something to do with the crons because the rest of the day i dont have any problems only in the morning, i dont know what todo

 

[IgorG]

Plesk is installed on a VPS

I'm not familiar with KVM but I'd suggest you check memory resources for your VPS. Or as help of Odin Support Team for checking this issue in detail directly on your server.

 

[omexlu12]

@IgorG

free -m and uptime dont show any strange on this time, hat what i dont understand.
How many this costs from Odin Support?

What can i do else ?

 

[IgorG]

How many this costs from Odin Support?

http://www.odin.com/support/buy-support/

 

[omexlu12]

Thanks but very expensive

@IgorG
EDIT: i have run the daily crons manually to see if thats the problem, but no 502 error:

cd / && run-parts --report /etc/cron.daily

I will now test to disable nginx and only use apache and see if the problem are still there :/

 

[trialotto]

@omexlu12,

Note the facts that

1 - it seems to be the case that only a specific cronjob (or two) causing a resource overload at a specific time,
2 - it can be expected that the whole issue has nothing to do with Apache, Nginx, Plesk etc.,
3 - you are running on a server, with a RAID10 setup, suggesting that resources are shared more than they ought to be.

From the above, one could or should conclude that (at least) the time, at which the cronjob is running, is the root cause of the problem.

In short, you have two simple options to consider:

a) change the time of the cronjob(s), by preference let them execute on a later hour,

Note that this is a "work-around", since it is very likely that the (main) server, on which your VPS is running, is running a lot of processes during the night, causing your cronjobs to result (indirectly) in a jam up and associated 502 error(s).

This "work-around" does not guarantee the solution of your problem: (on the one hand), it is very likely that the before mentioned (main) server does not have the resources to perform all tasks, including those on the VPS-es, and (on the other hand) it is a trial-and-error work-around, in the sense that you probably have to try over and over again.

In short, your Plesk installation is not the issue, the (main) server and VPS infrastructure is likely to be a danger to running Plesk on the VPS.

b) change the VPS and move to another provider.

To be realistic, it is very likely that all of the alleged (!) features of your VPS with netcup.de are overstated: the 1Gb/s backport is probably the backport of the main server (not the VPS), the 12GB RAM is very likely to be a partial swap memory, the SSD drives are RAID10 (implying that heavy loads in the form of disk writes/rewrites by other VPS will reduce the disk performance for your VPS) and so on and so on.

One can at least state for fact that it is totally ridiculous that your simple Plesk installation can not run properly on a 12GB Ram VPS.

In my humble opinion, the best advice would be: change the provider.

After all, you can change all you want, but sooner or later, the same or similar issues will arise.

Regards....