Resolved upstream timed out (110: Connection timed out) while connecting to upstream

[rhand]

Server operating system version

AlmaLinux 8.8 x86_64

Plesk version and microupdate number

Plesk Obsidian 18.0.54.4

Just had our VPS upgraded to 4GB RAM and two cores. The OS is Alma Linux. So expected things to start working more soothingly. Accessing the server via Dutch VPN so one close to server from a location abroad. I got this error multiple times now opening WordPress Dashboard main page:

1692#0: *1900 upstream timed out (110: Connection timed out) while connecting to upstream

and in the browser you can see

504 Gateway Time-out​

I also read Issue - NGNIX ERROR "upstream timed out (110: Connection timed out)" but already had set these settings to deal with time-outs:

FcgidIdleTimeout 120 FcgidProcessLifeTime 120 FcgidConnectTimeout 120 FcgidIOTimeout 120 Timeout 120 ProxyTimeout 120 FcgidIdleTimeout 80 FcgidProcessLifeTime 60 FcgidConnectTimeout 60 FcgidIOTimeout 90

This seems to be related to the upstream. We use PHP FM and Apache as a proxy server for Nginx. This seems to be the standard Plesk setup or the one done by TransIP.nl for our VPS.

Nginx config shows

location / { proxy_pass https://xx.xx.xxx.xxx:7081; proxy_hide_header upgrade; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Accel-Internal /internal-nginx-static-location; access_log off; }

Nginx is in proxy mode and uses Apache in front. But now I need to see how I can debug and fix this issue. Any ideas how?

 

[rhand]

When I checked for the upstream - see below - it seems that is Apache is the upstream with issues and does not refer to PHP FPM though it also needed more child processes so increased max to even 20 for now.

2023/09/25 03:34:47 [error] 1652#0: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxx.51.91.128, server: site.nl, request: "GET /wp-login.php HTTP/2.0", upstream: "https://xx.xx.xxx.180:7081/wp-login.php", host: "site.nl"
[root@example vhosts]# date
Mon Sep 25 03:40:51 CEST 2023
[root@example vhosts]# lsof -i tcp:443
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
^[[nginx 1645 root 15u IPv6 28519 0t0 TCP example.com:https (LISTEN)
nginx 1645 root 17u IPv4 28521 0t0 TCP xx-xx-xxx-180.colo.transip.net:https (LISTEN)
nginx 1652 nginx 15u IPv6 28519 0t0 TCP example.com:https (LISTEN)
nginx 1652 nginx 17u IPv4 28521 0t0 TCP xx-xx-xxx-180.colo.transip.net:https (LISTEN)
[root@example vhosts]# lsof -i tcp:7081
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
httpd 941 root 6u IPv6 26907 0t0 TCP *:7081 (LISTEN)
httpd 1674 apache 6u IPv6 26907 0t0 TCP *:7081 (LISTEN)
httpd 1681 apache 6u IPv6 26907 0t0 TCP *:7081 (LISTEN)
httpd 1682 apache 6u IPv6 26907 0t0 TCP *:7081 (LISTEN)
httpd 1684 apache 6u IPv6 26907 0t0 TCP *:7081 (LISTEN)
httpd 5627 apache 6u IPv6 26907 0t0 TCP *:7081 (LISTEN)

 

[rhand]

Testing

proxy_connect_timeout 180s;
proxy_send_timeout 180s;
proxy_read_timeout 180s;

after reading An operation or a script that takes more than 60 seconds to complete fails on a website hosted in Plesk: nginx 504 Gateway Time-out - Support Cases from Plesk Knowledge Base . Should help with longer time given for Nginx to connect to Apache. This besides these Apache settings I added and mentioned earlier

FcgidIdleTimeout 120
FcgidProcessLifeTime 120
FcgidConnectTimeout 120
FcgidIOTimeout 120
Timeout 120
ProxyTimeout 120
FcgidIdleTimeout 80
FcgidProcessLifeTime 60
FcgidConnectTimeout 60
FcgidIOTimeout 90

 

[rhand]

I can load things now but very slow loading pages list or products. And I see this error perhaps related to Mod Security

XMLHttpRequest cannot load https://site. nl/wp-admin/admin-ajax.php due to access control checks.

Either that or issues with theme or plugins, but do not recall having any. Source on this XMLHTTPRequest cannot load URL_HERE due to access control checks

 

[rhand]

I turned off WP Rocket for now. I also run all static files directly via Nginx and turned on Nginx caching. I may drop using Nginx as a proxy and just use it as the default web server. Too bad my thread has not been accepted yet, but forging ahead.

 

[rhand]

Also still seeing XMLHttpRequest cannot load Site.nl: Domeinnaam en webhosting - Ga online, begin nu! due to access control checks so perhaps still a Mod Security or Immunity issue, not sure.

 

[Peter Debik]

ModSecurity blocks are logged as error code 403 in error_log. It does not really look like a ModSecurity issue, but like a programming issue of the scripts used in the website. They simply do not respond fast and seem to run very long so that they occupy PHP-FPM processes. With a bit of traffic, all children will be occupied, new children cannot be spawned, hence the website runs into a timeout. You can easily check it by disabling Nginx on your server altogether (/usr/local/psa/admin/sbin/nginxmng --disable) and then checking how long Apache needs to return a response from a URL.

 

[rhand]

Just fired `/usr/local/psa/admin/sbin/nginxmng --disable` and TTFB byte is 16 seconds for a basic WordPress page About the company. Things were faster on shared hosting before.

curl -I https://snaakontwerp.nl/producten
HTTP/1.1 301 Moved Permanently
Date: Mon, 25 Sep 2023 08:13:28 GMT
Server: Apache
X-Powered-By: PHP/8.0.30
X-Redirect-By: WordPress
Location: https://snaakontwerp.nl/producten/
X-Powered-By: PleskLin
Content-Type: text/html; charset=UTF-8

 

[Peter Debik]

It's probably an issue with the website configuration, e.g. plugins or themes. You can test it by creating an empty, comparable site, e.g. a fresh Wordpress installation in a fresh subdomain.

 

[rhand]

Yes, that is something one always can do. Stop using your shop, start a basic one and test. Will however try to solve the issue with the shop running as much as possible.

Tried to install query monitor from the Dashboard but installing a plugin even timed out. I also see a tremendous amount of /user/sbin/httpd -DFOREGROUND processes using htop. That and decent amount of mysqld processes. And that on a site that hardly has any traffic. That was also the case with MOD Security turned on. Of the two cores I however have 90% available and of 4GB RAM I have 50% available and Swap. I allow for 15 PHP processes but mostly see only 3-4 firing. RAM in WP I did limit per process using

/* frontend memory limit: */
define ( 'WP_MEMORY_LIMIT', '64M' );
/* backend memory limit: */
define( 'WP_MAX_MEMORY_LIMIT', '256M');

And that should be fine. I removed all special settings for Apache and Nginx directories but kept WP Rocket turned off for now. I may turn off Advanced Custom Fields and see if that helps as well as a small piece of custom code, but must say I never had this much pain setting up a VPS.

 

[Peter Debik]

WP Rocket itself could be a problem, because it sometimes spiders all pages of a website to create its cache. If there are many pages, the tool itself becomes a burden on cpu load, because it continuously runs requests against the website. This doesn't speed the site up, but slows it down, because the server is kept busy with updating the cache. The speed advantage from the cache is minimal, but the disadvantage of cache maintenance can be quite high.

 

[rhand]

WP Rocket has never caused issues on custom VPS or shared hosting or WordPress hosting. But without it it seemed the WordPress installation started to at least function some. Mind blowing, but true. How can I turn off Immunify? Could be the cause of the error

XMLHttpRequest cannot load https://site. nl/wp-admin/admin-ajax.php due to access control checks.

Also returned

proxy_connect_timeout 180s;
proxy_send_timeout 180s;
proxy_read_timeout 180s;

as Plesk setup with Nginx Proxy cannot live without it. Though I even got

Gateway Timeout​

The gateway did not receive a timely response from the upstream server or application.

 

[rhand]

Also 101779 until 101992 are Apache processes now. 101992-101779 = 213 . Seems a lot. did a service httpd restart just now

 

[rhand]

Checking Plesk task manager I see this a lot also

2023-09-26 01:44:01	Event 'phys_hosting_update' for object with ID '9' (snaakontwerp.nl)
	/usr/local/psa/admin/bin/php -f /usr/local/psa/admin/plib/scripts/interface_async_executor.php -- EventListener /usr/local/psa/admin/plib/modules/composer/library/EventListener.php handleEvent admin

At Issue - Failure: Event 'phys_hosting_update' for object with ID '26' they mentioned this but I do not see how this tells me whether there is an issue or not.

Also a lot of Fail 2 Ban ips

2023-09-26 02:27:05,561 fail2ban.filter [850]: INFO [ssh] Found 112.213.120.30 - 2023-09-26 02:27:05
2023-09-26 02:27:06,059 fail2ban.filter [850]: INFO [plesk-postfix] Found 46.148.40.195 - 2023-09-26 02:27:05
2023-09-26 02:27:19,452 fail2ban.filter [850]: INFO [plesk-postfix] Found 46.148.40.76 - 2023-09-26 02:27:19
2023-09-26 02:27:21,854 fail2ban.filter [850]: INFO [plesk-postfix] Found 46.148.40.199 - 2023-09-26 02:27:21
2023-09-26 02:27:21,855 fail2ban.filter [850]: INFO [plesk-postfix] Found 46.148.40.58 - 2023-09-26 02:27:21
2023-09-26 02:27:38,521 fail2ban.filter [850]: INFO [plesk-postfix] Found 46.148.40.193 - 2023-09-26 02:27:38
2023-09-26 02:27:44,702 fail2ban.filter [850]: INFO [plesk-postfix] Found 46.148.40.49 - 2023-09-26 02:27:44
2023-09-26 02:28:08,493 fail2ban.filter [850]: INFO [plesk-postfix] Found 87.120.84.90 - 2023-09-26 02:28:08
2023-09-26 02:28:08,952 fail2ban.filter [850]: INFO [plesk-postfix] Found 46.148.40.145 - 2023-09-26 02:28:08
2023-09-26 02:28:22,513 fail2ban.filter [850]: INFO [plesk-postfix] Found 46.148.40.119 - 2023-09-26 02:28:22
2023-09-26 02:28:23,603 fail2ban.filter [850]: INFO [plesk-postfix] Found 46.148.40.65 - 2023-09-26 02:28:23
2023-09-26 02:29:14,740 fail2ban.filter [850]: INFO [plesk-postfix] Found 46.148.40.149 - 2023-09-26 02:29:14
2023-09-26 02:29:16,203 fail2ban.filter [850]: INFO [plesk-postfix] Found 46.148.40.68 - 2023-09-26 02:29:15
2023-09-26 02:29:17,953 fail2ban.filter [850]: INFO [plesk-postfix] Found 46.148.40.114 - 2023-09-26 02:29:17
2023-09-26 02:29:18,853 fail2ban.filter [850]: INFO [plesk-postfix] Found 46.148.40.191 - 2023-09-26 02:29:18

 

[rhand]

Also seems plugin installation fails and connection to WordPress.org is really slow

time wget WordPress.org
--2023-09-26 02:44:04-- Blog Tool, Publishing Platform, and CMS - WordPress.org
Resolving wordpress.org (wordpress.org)... 198.143.164.253
Connecting to wordpress.org (wordpress.org)|198.143.164.253|:80... failed: Connection timed out.
Retrying.

--2023-09-26 02:46:19-- (try: 2) Blog Tool, Publishing Platform, and CMS - WordPress.org
Connecting to wordpress.org (wordpress.org)|198.143.164.253|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: Blog Tool, Publishing Platform, and CMS - WordPress.org [following]
--2023-09-26 02:46:19-- Blog Tool, Publishing Platform, and CMS - WordPress.org
Connecting to wordpress.org (wordpress.org)|198.143.164.253|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'index.html'

index.html [ <=> ] 134.43K 521KB/s in 0.3s

2023-09-26 02:46:20 (521 KB/s) - 'index.html' saved [137660]


real 2m16.541s
user 0m0.126s
sys 0m0.045s

Perhaps I need to update name server:

cat /etc/resolv.conf
# Generated by NetworkManager
search com
nameserver 195.8.195.8
nameserver 2a01:7c8:7000:195:0:8:195:8
nameserver 2a01:7c8:7000:195:0:135:195:135

But not sure how yet. https://support.plesk.com/hc/en-us/...28-Resolving-timed-out-after-3512-millisecond mentions replacing ipv4 with Google's one, but nothing on first line nor ipv6 addresses.

 

[rhand]

I checked panel log and saw failed auto update, failed LE SSL for domain, but we did not add domain for Plesk yet. Already checked sites with Immunify and no viruses found.

tail -n 30 /var/log/plesk/panel.log

[2023-09-26 00:21:26.947] 92688:651207d2d94cb ERR [extension/wp-toolkit] Unable to retrieve data by "https://api.wordpress.org/core/stable-check/1.0/"
[2023-09-26 00:21:46.979] 92688:651207d2d94cb ERR [extension/wp-toolkit] Unable to retrieve data by "https://api.wordpress.org/core/stable-check/1.0/"
[2023-09-26 00:41:13.016] 93696:65120c82674aa ERR [extension/sslit] Unable to secure panel example.com automatically Invalid response from https://acme-v02.api.letsencrypt.org/acme/new-order.
Details:
Type: urn:ietfarams:acme:error:rejectedIdentifier
Status: 400
Detail: Error creating new order :: Cannot issue for "example.com": The ACME server refuses to issue a certificate for this domain name, because it is forbidden by policy

[2023-09-26 01:10:59.153] 94434:651211ab6949a ERR [extension/wp-toolkit] WP-CLI command has not finished working in 60 seconds, so it was terminated. Usually this means that there are problems with WordPress instance WordPress installation #6 ('https://staging.snaakontwerp.nl') itself, for example it could be infected with malware. Check the wp-config.php file of the instance for potential malware code.
[2023-09-26 01:11:00.293] 94434:651211ab6949a ERR [extension/wp-toolkit] Failed to reset cache for the instance #6, reason: Unable to finish running an operation on this site in 60 seconds. Operation was canceled. This is most likely caused by a server load spike, uncommon directives in wp-config.php, or a malware infection. Try running the operation again later. If this problem persists, make sure there are no customizations in wp-config.php that could slow down parsing this file, and consider checking the website for viruses and malware.
[2023-09-26 01:11:00.364] 94434:651211ab6949a ERR [extension/wp-toolkit] Unable to process auto updates for WordPress installation #6 ('https://staging.snaakontwerp.nl'): Failed to reset cache for the instance #6: Unable to finish running an operation on this site in 60 seconds. Operation was canceled. This is most likely caused by a server load spike, uncommon directives in wp-config.php, or a malware infection. Try running the operation again later. If this problem persists, make sure there are no customizations in wp-config.php that could slow down parsing this file, and consider checking the website for viruses and malware.
[2023-09-26 01:41:12.699] 96088:65121a9291515 ERR [extension/sslit] Unable to secure panel example.com automatically Invalid response from https://acme-v02.api.letsencrypt.org/acme/new-order.
Details:
Type: urn:ietfarams:acme:error:rejectedIdentifier
Status: 400
Detail: Error creating new order :: Cannot issue for "example.com": The ACME server refuses to issue a certificate for this domain name, because it is forbidden by policy

[2023-09-26 01:41:25.034] 95978:65121a95ceb6b ERR [extension/xovi] Sorry, currently not able to connect to API.
[2023-09-26 01:44:17.669] 99143:65121b4262d39 ERR [extension/xovi] Sorry, currently not able to connect to API.
[2023-09-26 01:48:44.070] 100942:65121c4cd7ec0 ERR [extension/xovi] Sorry, currently not able to connect to API.
[2023-09-26 02:00:43.579] 102874:65121f1c50d71 ERR [extension/xovi] Sorry, currently not able to connect to API.
[2023-09-26 02:36:07.090] 112324:651227681352c ERR [extension/xovi] Sorry, currently not able to connect to API.
[2023-09-26 02:41:12.592] 112795:651228a23c9f1 ERR [extension/sslit] Unable to secure panel example.com automatically Invalid response from https://acme-v02.api.letsencrypt.org/acme/new-order.
Details:
Type: urn:ietfarams:acme:error:rejectedIdentifier
Status: 400
Detail: Error creating new order :: Cannot issue for "example.com": The ACME server refuses to issue a certificate for this domain name, because it is forbidden by policy

[2023-09-26 02:42:53.830] 112851:651228fec6d97 ERR [extension/xovi] Sorry, currently not able to connect to API.

 

[Peter Debik]

Have you considered that the website scripts are simply the problem? From all you show above it is evident that the website is responding slow. It is probably a combination of bad bots hitting the site (or many regular visitors) while the site takes way too long to answer requests. That is a problem of the website. You can't cure it by trying to fix symptoms. A response timeout of 180 seconds for example is unrealistic. Noone will wait 3 minutes on a website to respond. If the site does not respond within seconds under heavy load, you'll need to fix the site, not the server.

 

[rhand]

The issue in the end was simple, but overlooked. The default Plesk firewall had all outgoing traffic blocked. This blocked api access to WordPress.org, functioning of WordPress heartbeat and so on. Once I turned off the Plesk firewall and only kept the TransIP firewall all was well. This thanks to server admin back in the Netherlands who helped me out and will be remembered for future issues. And so will everyone here in this thread who tried to help out. Thanks @Peter Debik for all your feedback.

 

[rhand]

Only WP Toolkit keeps on showing the site is in quarantine despite me refreshing. That is odd still.

 

[rhand]

Do see

[2023-09-28 13:00:18.283] 20248:65155c85938c4 ERR [extension/wp-toolkit] Failed to reset cache for the instance #6, reason: Unable to finish running an operation on this site in 60 seconds. Operation was canceled. This is most likely caused by a server load spike, uncommon directives in wp-config.php, or a malware infection. Try running the operation again later. If this problem persists, make sure there are no customizations in wp-config.php that could slow down parsing this file, and consider checking the website for viruses and malware.
[root@site]#