Forwarded to devs PHP 7.3 FPM dies without a trace

[H9k]

TITLE:

PHP 7.3 FPM dies without a trace​

PRODUCT, VERSION, OPERATING SYSTEM, ARCHITECTURE:

Plesk 17.8.11 Update35, Ubuntu 16.04 LTS, x64​

PROBLEM DESCRIPTION:

A couple of times per day, the fpm process for php 7.3 just dies, leaving all websites using PHP 7.3 down.
The watchdog does not detect any problems either.
There is no sign of the process dying in the fpm logs.​

STEPS TO REPRODUCE:

Have a couple of websites on PHP 7.3 FPM and wait...​

ACTUAL RESULT:

All PHP 7.3 websites dead, FPM process not running anymore​

EXPECTED RESULT:

I would expect the FPM process to not die in the first place, but also the watchdog to realize that it is down and start it automatically.​

ANY ADDITIONAL INFORMATION:

​

YOUR EXPECTATIONS FROM PLESK SERVICE TEAM:

Confirm bug​

 

[Achim Drees]

same here

 

[IgorG]

From developer:

Certainly, I need steps to reproduce.

In another similar request it is required to have ~35 sites. Support was unable to reproduce however. Please, try to ask for as much details as possible.

1. Is it real die (crash) or php master process stuck (a kind of hang, but master process may send alive messages to systemd). Even when apache or nginx report 5xx error codes, sometimes it possible to get response from php-fpm socket directly using e.g. /usr/bin/cgi-fcgi utility from libfcgi0ldbl packages. Previously I have seen reports concerning such states as "crashes" despite master process and some children were alive.
2. systemctl status, ps axuwf for php-fpm subtree if php-fpm process is present in the system, list of sockets open by php-fpm (lsof or netstat -xp / ss -xp).
3. systemctl status plesk-php73-fpm when php-fpm is dead for exit status and last executed command.
3. jornalctl -u plesk-php73-fpm
4. Site content, maybe last served requests (e.g. from domain access or error log). I hope to get scenario "install something and try to access some page".
5. Try to enable debug logs for plesk-php73-fpm
6. Is the issue related to reloads?

 

[H9k]

I just have about 8 websites configured on php 7.3

1. It's a real die, the process is not running anymore
2. systemctl status reports "failed", code "killed" and signal "SEGV"
3. journalctl reports no entries
4. site content is HTTP 503 service unavailable
5. will do
6. i don't think so, but not 100% sure

 

[Monty]

FYI: As a workaround (to have some peace over the holiday season) you can leverage the restart-on-failure feature from systemd, which will automatically restart a failed service. To implement that for the plesk-php73-fpm.service:
* mkdir /etc/systemd/system/plesk-php73-fpm.service.d
* create a file "plesk-php73-fpm.conf" in /etc/systemd/system/plesk-php73-fpm.service.d with the content:

[Service]
Restart=on-failure
RestartSec=5

* reload systemd using: systemctl daemon-reload

After that, plesk-php73-fpm will be restarted after 5 seconds in case of an unexpected crash (like segfault).

More info: systemd.service

 

[IgorG]

Since it is SIGSEGV (segmentation fault), is it possible to reproduce by accessing some specific page? That is why was asking for access logs.
Are there any specific PHP settings on affected domains (e.g. ACP, opcache, some extensions)? Is it some widespread application such as WordPress or specially developed site?
If is not a heavily loaded server, please, try to enable debug logs

log_level = debug

in /opt/plesk/php/7.3/etc/php-fpm.conf

 

[H9k]

I enabled debug logging, however it did not crash since... so I will post once it did.
I have no widespread application running, just some self made stuff using some libraries like smarty.

I checked the error logs and the only error that gets logged is a deprecation notice.

[Wed Dec 26 09:43:57.497043 2018] [proxy_fcgi:error] [pid 28380] [client 139.199.63.33:43304] AH01071: Got error 'PHP message: PHP Deprecated: The each() function is deprecated. This message will be suppressed on further calls in /var/www/vhosts/blabla.tld/httpdocs/lib/smarty/sysplugins/smarty_internal_compilebase.php on line 52'
[Wed Dec 26 10:02:03.460098 2018] [proxy_fcgi:error] [pid 25946] [client 46.163.75.107:36912] AH01071: Got error 'PHP message: PHP Deprecated: The each() function is deprecated. This message will be suppressed on further calls in /var/www/vhosts/blabla.tld/httpdocs/lib/smarty/sysplugins/smarty_internal_compilebase.php on line 52'
[Wed Dec 26 10:02:03.661697 2018] [proxy_fcgi:error] [pid 25943] [client 46.163.75.107:36934] AH01071: Got error 'PHP message: PHP Deprecated: The each() function is deprecated. This message will be suppressed on further calls in /var/www/vhosts/blabla.tld/httpdocs/lib/smarty/sysplugins/smarty_internal_compilebase.php on line 52'
[Wed Dec 26 10:06:34.717136 2018] [proxy:error] [pid 25947] (111)Connection refused: AH02454: FCGI: attempt to connect to Unix domain socket /var/www/vhosts/system/blabla.tld/php-fpm.sock (*) failed
[Wed Dec 26 10:06:34.798046 2018] [proxy_fcgi:error] [pid 25947] [client 35.184.73.193:50404] AH01079: failed to make connection to backend: httpd-UDS
[Wed Dec 26 10:08:00.176981 2018] [proxy:error] [pid 25947] (111)Connection refused: AH02454: FCGI: attempt to connect to Unix domain socket /var/www/vhosts/system/blabla.tld/php-fpm.sock (*) failed
[Wed Dec 26 10:08:00.177130 2018] [proxy_fcgi:error] [pid 25947] [client 23.103.64.39:52680] AH01079: failed to make connection to backend: httpd-UDS

However I dont see any direct relation between this error and the crash. Those deprecation errors also appeared in PHP 7.2 and that worked super stable for months. Anyway, I updated the Smarty libs and no more errores are being logged while browsing the websites. I will just wait for the next crash to happen and see if the debug logs reveal anything.

 

[H9k]

Hi there! I updated Smarty on the affected websites, so i had no more php errors/warnings on any of them.
Still, the FPM crashed twice today so far, ad 11:02 and 21:02.
The websites have very little traffic, but each hour at :02 the monitoring task runs, which just does a head request on those few domains just to see if they are alive.
I have attached a zip file containing the debug log from the FPM service, I hope it is helpful to find the cause of the problem.
I anonymized the domains in the log of course.

 

[Mike99]

Hi there, I can confirm, I have the same problem, PHP 7.3 service dying. Nothing special, standard Plesk (most recent updated version) setup without any additional extensions.

[75523.861269] php-fpm[1276]: segfault at 104 ip 000000000071036f sp 00007ffc77009160 error 4 in php-fpm[400000+404000]
[91126.800214] php-fpm[2082]: segfault at 104 ip 000000000071036f sp 00007ffe03179e60 error 4 in php-fpm[400000+404000]
[92784.477542] php-fpm[28093]: segfault at 104 ip 000000000071036f sp 00007ffd9849c7d0 error 4 in php-fpm[400000+404000]
[93613.944242] php-fpm[32587]: segfault at 104 ip 000000000071036f sp 00007ffce4b3f310 error 4 in php-fpm[400000+404000]
[93670.167511] php-fpm[4414]: segfault at 104 ip 000000000071036f sp 00007ffdad160650 error 4 in php-fpm[400000+404000]
[225606.181837] php-fpm[31135]: segfault at 104 ip 000000000071036f sp 00007ffd160907a0 error 4 in php-fpm[400000+404000]
[277805.841564] php-fpm[7639]: segfault at 104 ip 000000000071036f sp 00007ffcafc44ec0 error 4 in php-fpm[400000+404000]

Can I help providing some specific logs?

 

[lippoliv]

I have just one website using PHP7.3. WordPress 4.7, round about 1150 Posts, 65 Plugins including wooCommerce.

Since a week we updated to PHP7.3 and in that week it crashed 3 times without restart :/

Gesendet von meinem LG-H815 mit Tapatalk

 

[VGS]

Same here, PHP5.6 and PHP7.1 works perfect, but PHP7.3 crashes suddenly randomly.

Twice this morning and counting. So meanwhile, we are going to move to PHP7.2.

 

[lippoliv]

Same here, PHP5.6 and PHP7.1 works perfect, but PHP7.3 crashes suddenly randomly.

Twice this morning and counting. So meanwhile, we are going to move to PHP7.2.

I did the same. No problems since downgrading to PHP7.2

Gesendet von meinem LG-H815 mit Tapatalk

 

[Dukemaster]

Same here. Since christmas I got crazy by PHP 7.3. Everything the same like the others here in the topic.
only 8 domains (by administrator) on the server
4 domains with WoltLab community software (which supports 7.3)
Config: Only nginx (no apache) with FPM by nginx.
works perfect with PHP 7.2 (+7.1,+7.0) => only with FPM 7.3 the FPM crashes with no restart!

@IgorG feel free to check my server, I trust Plesk persons.

 

[cloudbridge]

Same here... With php 7.2 all fine, but 7.3 with apache or nginx dies without trace.

 

[IgorG]

Developers informed me that Plesk php-7.3.1 builds are coming. When this version released new debug builds will be prepared for support for performing deep investigations. So, you can contact the support team for this.
Additionally, they found that only workers were crashing, perhaps due to "accel_replace_string_by_process_permanent: Assertion `0' failed" that is closed with"
This seems to be fixed in PHP 7.3.1

 

[IgorG]

@Dukemaster if you have configured using 1and1 mirror repos for autoinstaller, most probably that php7.3.1 is not mirrored there yet.

 

[Dukemaster]

Thanks @IgorG
I was wrong. They are PLESK mirrors repos, not by 1and1.
I deleted my post before because few minutes later I realized that PHP 7.3.1 update came over PLESK PANEL UPDATES/UPGRADES.
But I was surprised that the problem mentioned here in this thread is already existing after updating to 7.3.1.
Wished that it perhaps would have been fixed with PHP 7.3.1.
But I'm sure your developers will do their best to fix it as soon as possible.

 

[H9k]

I upgraded to 7.3.1 but it still crashes for me unfortunately...

 

[Achim Drees]

same here

 

[Dukemaster]

Same too

Now it worked for incredible 32 hours, but 8 minutes ago FPM PHP 7.3.1. served by nginx was down again.
I thought and wished the bug was fixed by yesterdays PLESK updates.