Issue Apache reverse proxy error

[nisamudeen97]

Hi,

We have a 20 Core with 187Gb ram and 1TB SSD disk server on which we have setup Ubuntu 18 with plesk Obsidian. We are trying to achieve maximum through put. We are testing server by sending real requests using "loader.io", Website is configured to use php 7.2 fpm with nginx and below fpm settings.

pm = dynamic
pm.max_children = 5000
pm.start_servers = 50
pm.min_spare_servers = 20
pm.max_spare_servers = 70
pm.max_requests = 5000

Up on crossing 2.2 k incoming connections in server we are receiving / experiencing timeouts in loader.io, see the image below. On server logs i am not getting no errors related to nginx but on proxy error log (proxy_error_log), I can see two logs repeatedly logging once it touches 2.2K, both of the errors are related to apache reverse proxy. How we can solve the below errors and attain maximum through put?

Note:- There is no CPU or Ram usage impact on server on 2.2k real time requests. Cpu and ram usage is below 1%


2020/04/28 13:29:10 [error] 31954#0: *36723 connect() failed (110: Connection timed out) while connecting to upstream, client: 173.245.54.241, server:domain.com request: "GET /product-category/gender/male/?rng_min_pa_frame-width=45&rng_max_pa_frame-width=51&rng_order_pa_frame-width=ASC HTTP/1.1", upstream: "

https://server-ip:7081

/product-category/gender/male/?rng_min_pa_frame-width=45&rng_max_pa_frame-width=51&rng_order_pa_frame-width=ASC", host: "domain.com"

and


2020/04/28 13:29:10 [error] 31954#0: *30548 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream, client: 162.158.78.114, server: domain.com, request: "GET /product-category/gender/male/?rng_min_pa_frame-width=45&rng_max_pa_frame-width=51&rng_order_pa_frame-width=ASC HTTP/1.1", upstream: "https://server-ip:7081/product-category/gender/male/?rng_min_pa_frame-width=45&rng_max_pa_frame-width=51&rng_order_pa_frame-width=ASC", host: "domain.com"

 

[blamb]

pm.max_requests maybe can be updated. Have you tried lower values? e.g. below 100, maybe 50. 25, or 15? That recycles the sockets faster.

Also note that "static" is a better way to properly tune a server, so if set to static then calculate how much free memory the server has without php-fpm running, divide that by the memory the largest application needs, thats your max children you can have, set that to a conservative amount that comes in a tad under that.

Then, keep lowering the max_requests until the thing stops locking up with stale non timed-out sockets. Sometimes those locekd up socket timeouts are caused by connections that the applications don't properly close.


So for example:
#go static
pm = static
# if your scripts apprx max mem usage is 25MB, and you have 125GB of that mem you can throw at php-fpm, 125GB/25MB = 5000.
pm.max_children = 5000 (so your good there)
pm.max_requests = 100 # helps work around memory leaks


From here, you can just up the the max-children until you squeeze all the available mem out of server you want, then back off a tad to give it some headroom.

As far as CPU, the memory should be your bottleneck in my experience, so you may have to check what is using up so much CPU also, and correct that somehow as well.

 

[Bitpalast]

Three persons, four opinions ;-)
Here is mine: Please change "dynamic" to "ondemand". That will clear resources automatically.