Issue End of script output before headers / (104)Connection reset by peer: [client xx.xx.xx.xx:48] mod_fcgid: error reading data from FastCGI server

[maximus99]

Server operating system version

Centos7.9

Plesk version and microupdate number

18.0.52

Hi All,

I need an advise her as one of our customer is frequently getting 500 error , When ever he tried to run a deployment script pushing multiple api calls like every sec 2 calls almost in a min 100 to 120+ calls
Server is running with Cloudlinux 7.9 with Plesk obsidian 18.0.52.

And getting this below errors
[Wed Jan 24 12:13:01.803020 2024] [core:error] [pid 1222456:tid 140597936711424] [client xx.xx.xx.x8:5492] End of script output before headers: index.php
[Wed Jan 24 12:14:01.834975 2024] [fcgid:warn] [pid 1222113:tid 140597708916480] (104)Connection reset by peer: [client xx.xx.xx.x8:496] mod_fcgid: error reading data
from FastCGI server

to avoid this we have added the parameters to our domain of apache settings under plesk
FcgidMaxRequestsPerProcess 500
FcgidBusyTimeout 3600
FcgidMaxRequestLen 1073741824
FcgidProcessLifeTime 7200

And also we have increased the PHP setting for domain. Also we have enough RAM and CPU in the server.

memory_limit from 256M to 1024M
max_execution_time 60 to 600
max_input_time 150 to 220

But still we are getting the 500 error .

 

[Peter Debik]

Maybe these can help:
FcgidMaxProcesses 1000
FcgidMaxProcessesPerClass 40

 

[maximus99]

Hi Peter,

Will add those as well . Please confirm do i need to add along with FcgidMaxRequestsPerProcess ? as below

FcgidMaxRequestsPerProcess 500
FcgidBusyTimeout 3600
FcgidMaxRequestLen 1073741824
FcgidProcessLifeTime 7200
FcgidMaxProcesses 1000
FcgidMaxProcessesPerClass 40

 

[maximus99]

Getting this error when i tried to add FcgidMaxProcesses 1000 as Additional Apache directives
Invalid Apache configuration: AH00526: Syntax error on line 1 of /var/www/vhosts/system/example.com/conf/vhost_ssl.conf: FcgidMaxProcesses cannot occur within <VirtualHost> section

 

[maximus99]

Getting this error when i tried to add FcgidMaxProcesses 1000 as Additional Apache directives
Invalid Apache configuration: AH00526: Syntax error on line 1 of /var/www/vhosts/system/example.com/conf/vhost_ssl.conf: FcgidMaxProcesses cannot occur within <VirtualHost> section

do i need to add in under /etc/httpd/conf.d/fcgid.conf -- > FcgidMaxProcesses 1000 ?

 

[Peter Debik]

I should have said that my suggested variables need to be edited in /etc/httpd/conf.d/fcgid.conf, not in the virtual host section.

 

[maximus99]

Hi Peter,

Thanks for your suggestion , For now we are facing this for one particular customer out of more than 90+ customers hosted on that server.
Setting up that parameter in global under /etc/httpd/conf.d/fcgid.conf instead of that particular domain vhost . I hope it wont cause any impact on the other customers or do we need to increase the number from 1000 to much more ?

 

[Peter Debik]

1000 is already very high. You'd normally only need it on servers with lots of incoming web server requests. The values will normally not have negative effects.

 

[maximus99]

Hi Peter,

No luck after setting that parameter as we still get that 500 error . Please advise how to proceed further.

 

[Peter Debik]

How long do the processes you believe are causing the issue run?

 

[maximus99]

Lets say the customer run's deployment for the website . Every sec 2 api's will be called . Please find the below access logs for reference

Also all the request from the same IP address .
xx.xx.xx.xx - - [26/Jan/2024:08:45:24 +0100] "GET /api/sections/273 HTTP/1.0" 200
xx.xx.xx.xx- - [26/Jan/2024:08:45:24 +0100] "GET /api/sections/286 HTTP/1.0" 200
xx.xx.xx.xx - - [26/Jan/2024:08:45:25 +0100] "GET /api/sections/279 HTTP/1.0" 200
xx.xx.xx.xx - - [26/Jan/2024:08:45:25 +0100] "GET /api/sections/253 HTTP/1.0" 200
xx.xx.xx.xx - - [26/Jan/2024:08:45:26 +0100] "GET /api/sections/269 HTTP/1.0" 200
xx.xx.xx.xx- - [26/Jan/2024:08:45:26 +0100] "GET /api/sections/269 HTTP/1.0" 200
xx.xx.xx.xx - - [26/Jan/2024:08:45:27 +0100] "GET /api/sections/260 HTTP/1.0" 200
xx.xx.xx.xx- - [26/Jan/2024:08:45:27 +0100] "GET /api/courses/19 HTTP/1.0" 200
xx.xx.xx.xx- - [26/Jan/2024:08:45:28 +0100] "GET /api/sections/338 HTTP/1.0" 200
xx.xx.xx.xx- - [26/Jan/2024:08:45:29 +0100] "GET /api/courses/25 HTTP/1.0" 200
xx.xx.xx.xx - [26/Jan/2024:08:45:29 +0100] "GET /api/courses/19 HTTP/1.0" 200
xx.xx.xx.xx - - [26/Jan/2024:08:45:30 +0100] "GET /api/courses/1 HTTP/1.0" 200
xx.xx.xx.xx - - [26/Jan/2024:08:45:45 +0100] "GET /api/courses/1 HTTP/1.0" 200
xx.xx.xx.xx - - [26/Jan/2024:08:45:45 +0100] "GET /api/courses/15 HTTP/1.0" 200
xx.xx.xx.xx- - [26/Jan/2024:08:45:48 +0100] "GET /api/courses/15 HTTP/1.0" 200
xx.xx.xx.xx - - [26/Jan/2024:08:45:49 +0100] "GET /api/courses/33 HTTP/1.0" 200
xx.xx.xx.xx - - [26/Jan/2024:08:45:50 +0100] "GET /api/courses/33 HTTP/1.0" 200
xx.xx.xx.xx- - [26/Jan/2024:08:45:51 +0100] "GET /api/courses/2 HTTP/1.0" 200
xx.xx.xx.xx - - [26/Jan/2024:08:45:52 +0100] "GET /api/courses/2 HTTP/1.0" 500 ---- > 500 error
xx.xx.xx.xx- - [26/Jan/2024:08:45:53 +0100] "GET /api/courses/4 HTTP/1.0" 500 ---- > 500 error

One more thing i found in /var/log/httpd/error.log , frequently graceful restart
[Fri Jan 26 08:34:31 2024] [mpm_event:notice] [pid :tid 1104] AH00493: SIGUSR1 received. Doing graceful restart
[Fri Jan 26 08:34:572024] [mpm_event:notice] [pid :tid 1104] AH00493: SIGUSR1 received. Doing graceful restart
[Fri Jan 26 08:35:20. 2024] [mpm_event:notice] [pid :tid 1104] AH00493: SIGUSR1 received. Doing graceful restart
[Fri Jan 26 08:35:45. 2024] [mpm_event:notice] [pid :tid 1104] AH00493: SIGUSR1 received. Doing graceful restart
[Fri Jan 26 08:36:09. 2024] [mpm_event:notice] [pid :tid 1104] AH00493: SIGUSR1 received. Doing graceful restart
[Fri Jan 26 08:36:31. 2024] [mpm_event:notice] [pid :tid 1104] AH00493: SIGUSR1 received. Doing graceful restart

As we have the plesk web server settings --> "Apache restart interval" == 0 , "Once in a specified interval, Plesk checks for changes made to domains and subdomains. In case there are changes that require web server restarting, Plesk restarts Apache." . Will that because of this setting 0 interval as any changes made to domain or subdomain will bounce the httpd service might cause this 500 error ?

 

[Peter Debik]

An immediate restart is not reccommended. What can happen is that a restart or reload is requested while another one is not yet finished. That will for sure lead to problems. I suggest an interval of at least 600 seconds, maybe 900. It could indeed solve th issue.

However, I was more intrigued by the thought that the duration that your scripts run is too long. They might occupy a "slot" for a very long time so that after a certain number of requests all possible service slots are occupied and the server cannot process more requests. When you look at ps aux | grep php do you see long running PHP processes (excluding master processes)?

 

[maximus99]

We have tired to set that Apache restart interval from 300 to 1200 and still it didnt work .

That deployment script will run for almost 10 to 15 mins . And i didnt see any long php process holding up. As i observed max it will go to 250 and then it will drop down as you can see in the below count

~ ps -aux |grep php |wc -l
241
~ ps -aux |grep php |wc -l
245
~ ps -aux |grep php |wc -l
248
~ ps -aux |grep php |wc -l
57
~ ps -aux |grep php |wc -l
61
~ ps -aux |grep php |wc -l
89

 

[Peter Debik]

Please also try to count the number of httpd (RHEL, CentOS, Alma, Rocky) or apache2 (Debian, Ubuntu) processes. How many are there when the PHP number is high?

 

[maximus99]

over all Approximately 125+ httpd process

 

[Peter Debik]

I see. Probably, the overall number of concurrent processes is too high. If you can see 125 httpd processes, almos certainly at any time t there are more. The closer the number gets to 255, the slower, almost unresponsive the server becomes. The behavior is not linear, but exponential. For example when you reach 170 instances, you'll already experience a very slow server, some processes might run into timeouts. With 200+ you can almost certainly not get a response any longer.

~ ps -aux |grep php |wc -l
241

shows that it is very likely that at some point t you reach a situation where PHP or httpd time out. There is no configuration you can do on the server to prevent this but to lower the number of concurrently running processes.

 

[maximus99]

Ok , so you mean increasing the server resource like CPU or memory doesn't help here ? How much we increase the server resource we will face this issue ? Do we have any limit that only these much php or httpd process to run according to plesk ?
Do we have any options to fine tune it ? as we already has multiple customers in it

 

[Peter Debik]

This is a limit by the web server, not Plesk. It is hard coded in Apache. You will experience it in any environment.

Please think about a different approach for how you run your software. It cannot be right to start long running processes so that they overlap each other and create a mountain of processes that all need to be handled at the same time. What kind of application would do that? It must be possible to design the software in a different way.