Forwarded to devs Update hung with PHP 5.4.16 by OS vendor FPM application disabled

[Sergio Manzi]

TITLE:

Update hung with PHP 5.4.16 by OS vendor FPM application disabled​

PRODUCT, VERSION, OPERATING SYSTEM, ARCHITECTURE:

Plesk 17.8.11#1
CentOS 7.4.1708, Kernel 3.10.0-693​

PROBLEM DESCRIPTION:

please note: a complete account of what happened is available in the Issue - HELP! Update stuck with Plesk Onyx 17.8.11#1 thread

In my server I had all "PHP 5.4.xx by OS vendor" interfaces disabled. It's an old version and I don't want my clients be able to select it. Plesk too flags it as outdated.

I received an email from my server stating that 23 updates were available. Amongst them there were also upgrades for the OS PHP:

- php 5.4.16-43.el7_4.1 from updates repo (currently installed version: 5.4.16-43.el7_4 from updates repo)
- php-cli 5.4.16-43.el7_4.1 from updates repo (currently installed version: 5.4.16-43.el7_4 from updates repo)
- php-common 5.4.16-43.el7_4.1 from updates repo (currently installed version: 5.4.16-43.el7_4 from updates repo)
- php-fpm 5.4.16-43.el7_4.1 from updates repo (currently installed version: 5.4.16-43.el7_4 from updates repo)
- php-gd 5.4.16-43.el7_4.1 from updates repo (currently installed version: 5.4.16-43.el7_4 from updates repo)
- php-mbstring 5.4.16-43.el7_4.1 from updates repo (currently installed version: 5.4.16-43.el7_4 from updates repo)
- php-mysql 5.4.16-43.el7_4.1 from updates repo (currently installed version: 5.4.16-43.el7_4 from updates repo)
- php-pdo 5.4.16-43.el7_4.1 from updates repo (currently installed version: 5.4.16-43.el7_4 from updates repo)
- php-xml 5.4.16-43.el7_4.1 from updates repo (currently installed version: 5.4.16-43.el7_4 from updates repo)

I proceeded with an upgrade from the Plesk panel (/admin/pum/updates-list).

After waiting more than half an hour (I had a pop-up stating "1 Tasks in progress... - Updating 23 packages") I checked the process status through ssh:

psaadm   12133  0.0  1.1 272888 22168 ?        Ss   04:11   0:00 /usr/bin/sw-engine -c /usr/local/psa/admin/conf/php.ini /usr/local/
root     12137  0.0  0.0  28464  1788 ?        S    04:11   0:00  \_ /usr/local/psa/admin/bin/pum --update --json -- cloud-init ipta
root     12138  0.5  9.4 963544 178240 ?       S    04:11   0:08      \_ /usr/bin/python -Estt /usr/local/psa/admin/sbin/pum_worker
root     15934  0.0  0.0  11636  1340 ?        S    04:15   0:00          \_ /bin/sh /var/tmp/rpm-tmp.FNIFfk 1
root     15952  0.0  0.0  25236  1812 ?        S    04:15   0:00              \_ systemctl try-restart php-fpm.service

Tried restarting php-fpm manually:

# systemctl restart php-fpm
Job for php-fpm.service failed because the control process exited with error code. See "systemctl status php-fpm.service" and "journalctl -xe" for details.

# systemctl -l status php-fpm.service
● php-fpm.service - The PHP FastCGI Process Manager
   Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; disabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/php-fpm.service.d
          └─limit_nofile.conf
   Active: failed (Result: exit-code) since Fri 2018-03-09 04:40:12 GMT; 6min ago
  Process: 19100 ExecStart=/usr/sbin/php-fpm --nodaemonize (code=exited, status=78)
 Main PID: 19100 (code=exited, status=78)

Mar 09 04:40:12 ams301.smz.it systemd[1]: Starting The PHP FastCGI Process Manager...
Mar 09 04:40:12 ams301.smz.it php-fpm[19100]: [09-Mar-2018 04:40:12] ERROR: No pool defined. at least one pool section must be specified in config file
Mar 09 04:40:12 ams301.smz.it php-fpm[19100]: [09-Mar-2018 04:40:12] ERROR: failed to post process the configuration
Mar 09 04:40:12 ams301.smz.it php-fpm[19100]: [09-Mar-2018 04:40:12] ERROR: FPM initialization failed
Mar 09 04:40:12 ams301.smz.it systemd[1]: php-fpm.service: main process exited, code=exited, status=78/n/a
Mar 09 04:40:12 ams301.smz.it systemd[1]: Failed to start The PHP FastCGI Process Manager.
Mar 09 04:40:12 ams301.smz.it systemd[1]: Unit php-fpm.service entered failed state.
Mar 09 04:40:12 ams301.smz.it systemd[1]: php-fpm.service failed.

Please note the "ERROR: No pool defined" above...

To get out of the situation I had to kill the PUM process, but that left yum and PUM in a miserable state (see the above cited thread for details)
​

STEPS TO REPRODUCE:

• Reproduce the initial conditions described above
• Perform the upgrade

ACTUAL RESULT:

• Stuck upgrade
• Broken yum status

EXPECTED RESULT:

• A correct upgrade.

ANY ADDITIONAL INFORMATION:

I'm inclined to think that on upgrades involving php-fpm (OS Version) PUM tries to start a service that it should not start if that PHP version is disabled

​

YOUR EXPECTATIONS FROM PLESK SERVICE TEAM:

Confirm bug
Make upgrade process more robust and fault tolerant​

 

[Bitpalast]

Different reason, but same result with duplicate libraries and an incorrectly installed upgrade from 12.5 to 17.5 was reported by me in support ticket #35184 on April 29th, 2017. To me it seems that big problems always come into play when the initial upgrade attempt fails.

We have had several difficulties with upgrades that were not reported in a ticket, too. I can only support that the upgrade process should get a lot more attention by developers. We still have two 12.5 servers online, because we do not dare the upgrade as it will almost for sure break the system configuration. It would be great if there was a "staging" function, maybe a kind of virtualization of the upgrade process so that it can be tested before the actual upgrade is done.

 

[IgorG]

Developers can't reproduce this issue. Investigation directly on your server is required. So, please contact Plesk Support Team.
Thanks.

 

[Sergio Manzi]

I don't think accessing my server now will be revealing of anything: the system has already been upgraded to the latest "PHP-FPM by OS", something which I could achieve only by activating it in the PHP settings.

If they need logs (maybe traces of the failure are there...) I will be glad to provide: just ask for them!

If they really need to access my server, this can only be done through a Teamviewer session to my workstation and from there to the server: direct access to the server is out of question, sorry.

I think I can also be able to set-up a test VPS on DO to reproduce the issue (and I could provide access to that...): can you please point me on how to get a temporary Plesk license for that?

 

[IgorG]

can you please point me on how to get a temporary Plesk license for that?

How to get a trial license for Plesk?

 

[Sergio Manzi]

Thanks @IgorG !

I've setup a test machine with a trial license and I'm still confident I can duplicate the issue (what a stubborn old man I am!), but I'm facing a problem:

during the setup process my "PHP-FPM by OS vendor" has been already updated to the latest version. To duplicate the issue I should have "PHP-FPM by OS vendor" disabled in the PHP Settings and... an update available, which at this point I have not.

Is there a way to downgrade the current "PHP by OS vendor" version so that afterward Plesk will propose an upgrade?

TIA for any help you could provide!

 

[Sergio Manzi]

I found how to downgrade my test VPS to the previous "PHP-FPM by OS vendor:

yum downgrade php\*-5.4.16-43.el7_4

After that:

• I took a snapshot of my VPS, so that I can restart from that point in time.
• Plesk listed the new "php 5.4.16-43.el7_4.1" packages as available
• I had already disabled "PHP-FPM by OS vendor" in the plesk panel
• I proceeded upgrading from the Plesk panel exactly as I did before on my "live" VPS
• Everything went fine
• WTF!
• Can't reproduce...
•