Resolved Only some requests receiving 502 Bad Gateway

[varun33]

Issue
Some users are getting a 502 error and the error does not go away with a refresh of the page. Once that error pops-up any page requested on the domain throws the same 502 error. There is no specific website or domain from the ones configured on the server where this error is being faced it is just random. It only goes away after deleting the wordpress cookie that starts with 'wordpress_logged_in_*******' and 'wordpress_sec_*******'

Errors in the log
(Apache/Nginx)
[error] 60407#0: *31100 upstream prematurely closed connection while reading response header from upstream, client: xx.xx.xx.xx, server: [servername], request: "GET / HTTP/2.0", upstream: "https://xx.xx.xx.xx:7081/", host: [servername]

Server Environment
Ubuntu 18.04.1 LTS
CPU: 1 processor with 16 cores
RAM: 48G
Plesk Onyx Version 17.8.11 Update #36
PHP 7.2.14
FPM application served by Apache
SSL enabled on all domains
Nginx Proxy mode enabled along with smart static files processing
Wordpress multisite @ Linux+Nginx+Apache2+PHP-FPM-7.2+HTTP2+MySQL (mariadb)
15 domains configured with the above settings all running under a common wordpress multisite installation

Background
We recently migrated to the above server, before this we were using PHP 5.6.33 with FastCGI and did not have any such issues. Connected to Issue - Only some web requests getting 503 error

Our analysis
- We have tried all solutions on the internet for nginx configuration that suggest we increase the fastcgi_buffers or the proxy_buffers but none of them worked for us.
- No issues with the availability of CPU/memory as the average load is around 3-4 and there is at least 24G of memory available
- We can replicate the issue by re-using any wordpress authentication cookie('wordpress_logged_in_*******' OR 'wordpress_sec_*******') that has already faced this problem
- We have found that on the web browser if the issue occurs we can delete the wordpress cookie 'wordpress_logged_in_*******' OR 'wordpress_sec_*******' and that fixes the issue and allows to open the requested page instead of a 502 error page

We are still trying to find the cause for this issue but if any of you have faced this or a similar problem please share your version and the solution if you were able to find it.

Thank you.

 

[Bitpalast]

If this is linked to a Cookie being present or not, it is likely that the issue is caused by a Cache, e.g. Cloudflare caching or maybe Nginx caching, in conjunction with search engine optimized URLs. The problem in this case is that a dynamic function is identified as a static page by the cache, so that the expected evaluation of data does not take place, the function does not return a result within n seconds so that the front end proxy (Nginx) does not get a response or an empty response, hence it delivers the error message "upstream prematurely closed connection ...". You could try to disable search engine friendly URLs in wordpress and try again if it works after that change. You can also try to disable caches to verify if this is the cause.

 

[varun33]

We finally managed to resolve the issue. The problem was with the wordpress plugin "iThemes Security", due to a bug in the plugin related to expired wordpress login keys there was an infinite loop being created which ultimately lead to the 502/503 error.

Thank you all for your help.