Issue Passenger issue - boost::thread_resource_error: Resource temporarily unavailable

[Ming]

Hi,

i am runing a virtual machine with "Plesk Obsidian Version 18.0.41" on Ubuntu 20.04.4 LTS.

I have passenger installed to run OpenProject.

Somehow the performance is really bad and i get errors.

In the log file i can see some resource errors (/var/log/apache2/error.log)

[ N 2022-03-08 00:00:43.9974 344600/T1 age/Wat/WatchdogMain.cpp:1373 ]: Starting Passenger watchdog...
[ N 2022-03-08 00:00:44.0636 344603/T1 age/Cor/CoreMain.cpp:1340 ]: Starting Passenger core...
[ N 2022-03-08 00:00:44.0638 344603/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2022-03-08 00:00:44.0980 344603/T1 age/Cor/CoreMain.cpp:1015 ]: Passenger core online, PID 344603
[Tue Mar 08 00:00:44.161230 2022] [ssl:warn] [pid 166471:tid 140525254335552] AH01909: default:443:0 server certificate does NOT include an ID which matches the server name
[Tue Mar 08 00:00:44.161808 2022] [mpm_event:notice] [pid 166471:tid 140525254335552] AH00489: Apache/2.4.41 (Ubuntu) mod_fcgid/2.3.9 Phusion_Passenger/6.0.12 OpenSSL/1.1.1f configured -- resuming normal operations
[Tue Mar 08 00:00:44.161833 2022] [core:notice] [pid 166471:tid 140525254335552] AH00094: Command line: '/usr/sbin/apache2'
[ N 2022-03-08 00:00:50.0828 344603/T7 age/Cor/SecurityUpdateChecker.h:519 ]: Security update check: no update found (next check in 24 hours)
[Tue Mar 08 03:56:52.557626 2022] [cgi:error] [pid 344629:tid 140525046802176] [client x.x.x.x:y] AH02811: script not found or unable to stat: /var/www/vhosts/default/cgi-binindex2.asp, referer: http://x.x.x.x:
80/cgi-bin/index2.asp
[Tue Mar 08 06:13:21.410960 2022] [core:error] [pid 344629:tid 140525080372992] [client x.x.x.x:y] AH00126: Invalid URI in request GET /cgi-bin/.%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/host
s HTTP/1.1
[Tue Mar 08 10:53:32.314469 2022] [core:error] [pid 344629:tid 140525155907328] [client x.x.x.xy:y] AH00126: Invalid URI in request POST /cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh HTTP/1.1
App 370966 output: [passenger_native_support.so] not compiling because PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0
App 370966 output: [passenger_native_support.so] not downloading because PASSENGER_DOWNLOAD_NATIVE_SUPPORT_BINARY=0
App 370966 output: [passenger_native_support.so] will not be used (can't compile or download)
App 370966 output: --> Passenger will still operate normally.
App 372114 output: [passenger_native_support.so] not compiling because PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0
App 372114 output: [passenger_native_support.so] not downloading because PASSENGER_DOWNLOAD_NATIVE_SUPPORT_BINARY=0
App 372114 output: [passenger_native_support.so] will not be used (can't compile or download)
App 372114 output: --> Passenger will still operate normally.
terminate called after throwing an instance of 'Passenger::SystemException'
what(): Cannot fork() a new process: Resource temporarily unavailable (errno=11)
ERROR: cannot fork a process for executing 'tee'

[ pid=344603, timestamp=1646761144 ] Process aborted! signo=SIGABRT(6), reason=SI_TKILL, signal sent by PID 344603 with UID 0, si_addr=0x5421b, randomSeed=1646694044
[ pid=344603 ] Crash log files will be dumped to /var/tmp/passenger-crash-log.1646761144.CRnqi5 <--- ******* LOOK HERE FOR DETAILS!!! *******
[ pid=344603 ] Could not fork a child process for dumping diagnostics: fork() failed with errno=11
[ W 2022-03-08 18:39:04.9338 344600/T4 age/Wat/AgentWatcher.cpp:94 ]: Passenger core (pid=344603) crashed with signal SIGABRT, restarting it...
[ N 2022-03-08 18:39:05.9515 372279/T1 age/Cor/CoreMain.cpp:1340 ]: Starting Passenger core...
[ N 2022-03-08 18:39:05.9519 372279/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2022-03-08 18:39:06.0103 372279/T1 age/Cor/CoreMain.cpp:1015 ]: Passenger core online, PID 372279
App 372338 output: [passenger_native_support.so] not compiling because PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0
App 372338 output: [passenger_native_support.so] not downloading because PASSENGER_DOWNLOAD_NATIVE_SUPPORT_BINARY=0
App 372338 output: [passenger_native_support.so] will not be used (can't compile or download)
App 372338 output: --> Passenger will still operate normally.
[ N 2022-03-08 18:39:08.1801 372279/T5 age/Cor/SecurityUpdateChecker.h:519 ]: Security update check: no update found (next check in 24 hours)
[ E 2022-03-08 18:39:35.1610 372279/Tn age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /opt/openproject: An internal error occurred while spawning an application process: boost::thread_resource_
error: Resource temporarily unavailable
Error ID: d59f2d1e
Error details saved to: /tmp/passenger-error-FuTlIc.html

[ E 2022-03-08 18:39:35.1791 372279/Tf age/Cor/Con/CheckoutSession.cpp:281 ]: [Client 4-1] Cannot checkout session because a spawning error occurred. The identifier of the error is d59f2d1e. Please see earlier logs for detai
ls about the error.
[ E 2022-03-08 18:39:35.1792 372279/T9 age/Cor/Con/CheckoutSession.cpp:281 ]: [Client 1-2] Cannot checkout session because a spawning error occurred. The identifier of the error is d59f2d1e. Please see earlier logs for detai
ls about the error.

In the error html from passenger i got this message:
boost::thread_resource_error: Resource temporarily unavailable

So i check the resources.
The command "ulimit -a" gives me

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1545096
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 62987
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

After several researches i found this article https://support.plesk.com/hc/en-us/...-on-a-Plesk-server-Unable-to-fork-new-process

so i have change my mpm_event.conf

more mpm_event.conf
# event MPM
# StartServers: initial number of server processes to start
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestWorkers: maximum number of worker threads
# MaxConnectionsPerChild: maximum number of requests a server process serves

<IfModule mpm_event_module>
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestWorkers 256
ServerLimit 256
MaxConnectionsPerChild 0
</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Here, the strange things are that i have increase the number for MaxRequestWorkers from 150 to 256 instead of decreasing as mentioned in the article. Also the parameter ServerLimit was not there and i add it to the config. So maybe not my root cause.... ;-(
There was also in that directory a mpm_worker.conf which have similar parameters, which i haven't touch.

Can somebody help me out or have an idea/tip?

Thanks in advance

Best regards
Ming

 

[Nikolay]

Hi, can you tell how many resources your server has? Just to see a full picture

 

[Ming]

Hi Nikolay,
of course.

Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (6 core(s))
16.00 GB of RAM (13.88 GB free)

 

[Nikolay]

Thanks.
The error is generic, it is no clear what is the root cause, if this is the server or application itself.
I can suggest the following:
Apache:

set the value of MaxConnectionsPerChild to zero
set the value of MaxSpareThreads to the same value as MaxRequestWorkers

nginx:

Configuration reference - Nginx - Passenger Library

nginx config for passenger - /etc/nginx/conf.d/phusion-passenger.conf

Also, it creates a crashdump and it makes sense to check for any errors there. Crash dump path from the previous log

[ pid=344603 ] Crash log files will be dumped to /var/tmp/passenger-crash-log.1646761144.CRnqi5 <--- ******* LOOK HERE FOR DETAILS!!! *******
[ pid=344603 ] Could not fork a child process for dumping diagnostics: fork() failed with errno=11

 

[Ming]

Hi,
thanks for the answer.
the only logs i found related to it was this one (html dump)





When i check /etc/nginx/conf.d/phusion-passenger.conf there are no value for MaxRequestWorkers

more /etc/nginx/conf.d/phusion-passenger.conf

passenger_root /usr/share/passenger/phusion_passenger/locations.ini;

# Avoid downloading of sources or binary for ruby passenger_native_support.so
# after each nginx restart by default because of development packages
# or libruby.so.X.Y may be missed anyway.
# Ruby passenger_native_support.so is packaged for system ruby only.
passenger_env_var PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY 0;
passenger_env_var PASSENGER_DOWNLOAD_NATIVE_SUPPORT_BINARY 0;

Do you know which value to be set for MaxSpareThreads? And shall i also set MaxRequestWorkers?

I found also this -> Optimizing Passenger + Nginx - Passenger Library
It mentioned also to set passenger_min_instances.

And 2 last questions.

I set the parameters here /etc/nginx/conf.d/phusion-passenger.conf not here /etc/apache2/mods-available/passenger.conf, correct?

Do you know the correct syntax in /etc/nginx/conf.d/phusion-passenger.conf?

passenger_env_var MaxConnectionsPerChild 0;

or

passenger_env_var passenger_max_pool_size 6;

Thanks for your Feedback in advance.

Best regards

Ming

 

[Ming]

btw. i cannot found these 2 parameters MaxConnectionsPerChild and MaxSpareThreads here Configuration reference - Nginx - Passenger Library
Do you mean
MaxConnectionsPerChild = passenger_max_pool_size
and
MaxSpareThreads = ?

also check with max_spare_threads and max_connections_per_child

 

[mow]

Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (6 core(s))
16.00 GB of RAM (13.88 GB free)

That's a 12-core CPU, or 24 virtual cores with HT. Are those 6 real or 6 virtual cores?
Could it be that the machine is overbooked?

 

[Nikolay]

sorry for the delay.
The values below are for Apache web server

set the value of MaxConnectionsPerChild to zero
set the value of MaxSpareThreads to the same value as MaxRequestWorkers

For troubleshooting purposes try to increase it high and if it helps, late you can gradually decrease the value.

MaxConnectionsPerChild 4500
MaxSpareThreads to the same value as MaxRequestWorkers or 250

Configuration reference - Nginx - Passenger Library - this one does not present in default config, so you need to add it manually

Also, I would check the application itself. Try to deploy some simple app like 'Hello world!' and check how it goes.
If the application you are trying to launch is some 3rd party, review its requirements and make sure that all necessary components are installed on the server.

Running 'lscpu' can help to find more info about CPU

 

[Ming]

That's a 12-core CPU, or 24 virtual cores with HT. Are those 6 real or 6 virtual cores?
Could it be that the machine is overbooked?

Hi mow,

thanks for your Feedback. It shall be vCore

lscpu gives me the follow:

Architektur: x86_64
CPU Operationsmodus: 32-bit, 64-bit
Byte-Reihenfolge: Little Endian
Adressgrößen: 46 bits physical, 48 bits virtual
CPU(s): 6
Liste der Online-CPU(s): 0-5
Thread(s) pro Kern: 1
Kern(e) pro Socket: 6
Sockel: 1
Anbieterkennung: GenuineIntel
Prozessorfamilie: 6
Modell: 63
Modellname: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
Stepping: 2
CPU MHz: 1999.918
BogoMIPS: 4993.74
Virtualisierung: VT-x
Hypervisor-Anbieter: Parallels
Virtualisierungstyp: Container
L1d Cache: 192 KiB
L1i Cache: 192 KiB
L2 Cache: 1,5 MiB
L3 Cache: 30 MiB
Markierungen: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mc
a cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss
ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arc
h_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
aperfmperf eagerfpu cpuid_faulting pni pclmulqdq dtes64
monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtp
r pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_d
eadline_timer aes xsave avx f16c rdrand lahf_lm abm epb
invpcid_single intel_ppin ssbd ibrs ibpb stibp tpr_sha
dow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1
avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_o
ccup_llc dtherm ida arat pln pts md_clear spec_ctrl int
el_stibp flush_l1d

How can i check if CPU is overbooked, and what is HT?

 

[Ming]

sorry for the delay.
The values below are for Apache web server

set the value of MaxConnectionsPerChild to zero
set the value of MaxSpareThreads to the same value as MaxRequestWorkers

For troubleshooting purposes try to increase it high and if it helps, late you can gradually decrease the value.

MaxConnectionsPerChild 4500
MaxSpareThreads to the same value as MaxRequestWorkers or 250

Configuration reference - Nginx - Passenger Library - this one does not present in default config, so you need to add it manually

Also, I would check the application itself. Try to deploy some simple app like 'Hello world!' and check how it goes.
If the application you are trying to launch is some 3rd party, review its requirements and make sure that all necessary components are installed on the server.

Running 'lscpu' can help to find more info about CPU

Hi Nikolay,
thanks for the response.
I have try to set these values in different location like
/etc/nginx/conf.d/phusion-passenger.conf
or
/etc/apache2/mods-available/passenger.conf

with no effect.

Since you mention the values are for "The values below are for Apache web server", i also try to set it in Plesk
Domains -> Settings for Apache & nginx

SetEnv MaxConnectionsPerChild 0
SetEnv MaxSpareThreads 2500
SetEnv MaxRequestWorkers 2500

But still get the errors. I got the impression that the values are not taken.

Is there a possibility that i can check or see if these parameters are really taken and used?

Thanks in advance for your Feedback

Best regards

Ming

 

[mow]

Is there a possibility that i can check or see if these parameters are really taken and used?

nginx -T

 

[Ming]

wow very fast reply

nginx -T gives me command not found

 

[Ming]

strange seems like nginx is not installed... but i have this config file dir /etc/nginx/conf.d/phusion-passenger.conf