Resolved Failed Update to 18.0.37 on a default installation

[Bitpalast]

...
Error: Cannot open file /var/cache/yum/x86_64/7/PLESK_17_PHP74/packages/plesk-php74-pear-1.10.12-1centos.7.210730.1059.noarch.rpm: [Errno 2] No such file or directory: '/var/cache/yum/x86_64/7/PLESK_17_PHP74/packages/plesk-php74-pear-1.10.12-1centos.7.210730.1059.noarch.rpm'
TypeError: an integer is required
FATAL ERROR: python callback <bound method RPMTransaction.callback of <yum.rpmtrans.RPMTransaction instance at 0x7f7b579eee18>> failed, aborting!
processTransaction event: 10 (Downloading Packages)
processTransaction event: 11 (unknown)
processTransaction event: 20 (Check Package Signatures)
processTransaction event: 30 (Running Test Transaction)
processTransaction event: 40 (Running Transaction)
Bootstrapper has finished action (exec time: 0 sec.): parent_name='PLESK_18_0_37', sequence='pkgs', stage='rollback', sequence_order='1', operation='install', exec_cmd='rm -f /tmp/pp-bootstrapper-mode.flag; rm -f /var/lock/parallels-panel-maintenance-mode.flag; touch /var/lock/parallels-panel-upgrade-failure.flag; :'', m_arch='', exit code: 0, output: ~empty

I tried to create a support ticket and bought a subscription for it, but although I bough the subscription, the support ticket cannot be created. Now this is a really bad thing. [Edit: After a few minutes of wait time, probably for payment processing, the ticket could be created.]

Dovecot is not working correctly after this event.

 master: Error: service(pop3-login): command startup failed, throttling for 60.000 secs
Aug 03 09:13:52 <user> dovecot[28545]: imap: Fatal: Dovecot version mismatch: Master is v2.3.14, imap is v2.3.15 (if you don't care, set version_ignore=yes)

 

[Bitpalast]

I am currently trying to run plesk repair installation, but this is hanging at "Plesk database scheme upgrade has been completed".

 

[Bitpalast]

After a long wait on the completion of "plesk repair installation" all services seem to be operable again. There have been several errors during execution of the repair routine that are now being analyzed by support.

 

[Bitpalast]

The root cause of the failed Plesk update is that during the update procedure some cache files were missed at the Yum Cache folder located at /var/cache/yum/x86_64. Why it has happened on this one machine while other machines with identical Plesk installations were not affected so far, is unknown.

The solution was to
1) # plesk installer --select-release-current --reinstall-patch --upgrade-installed-components
(This responded that the system is up to date, no issues, which we know is not true.)

2) # plesk repair installation
(This lasted for half an hour to finish. One issue during the script run was that the web server configuration files are removed an a long wait later re-created. So this procedure will definitely halt web server operations for many minutes. On the specific machine we have 600 domains.) The command through many errors, but after all, all services were reconfigured correctly and could be restarted.

3) Afterwards - due to the failed initial upgrade - many duplicate packages were reported. On the "components" page of Plesk, the older versions 18.0.36 were shown while Plesk itself reported that it is running als 18.0.37.
# package-cleanup --dupes
The duplicates have been removed with the procedure explained in Upgrade to Plesk Obsidian or update between Obsidian releases fails: TypeError: an integer is required

It remains unclear why files were missing during installation.

 

[moswak]

did you initiate the update to 18.0.37 manually or was it an automatic update?
Scary when such an effort arises from an update.

 

[Bitpalast]

It was an auto-upgrade in the middle of the night, beyond control.

We have a lot of surveillance on the systems, but as the services did not fail and websites remained accessible, the surveillance did not notice it. For example, Dovecot remained active, but mailboxes could still not be accessed.

I agree that this is absolutely scary. I still don't know what preparations should have been made to avoid the "cache" issue. It has not occured on any other host yet, and all our host systems are identical setups with identical hardware and operating system version.

 

[Monty]

@Peter Debik Thanks for sharing! We did not (yet) have this issue on any of our servers but we'll keep a close eye on it.
I'm wondering if daily "yum clean all" would be a valid workaround for this...

 

[Bitpalast]

According to support, Plesk cannot control this behavior, because Yum cache is responsible for it. I am not sure what measures we'll need to take with Yum to avoid this issue. There are at least four commands. Maybed # yum clean all as @Monty suggests is the best way to go, but could this have side effects?
# yum clean packages
# yum clean metadata
# yum clean headers
# yum clean all
And why, if such issues are known, would the Plesk installer not clean this before they try to install new packages?

 

[Monty]

Just a (quite) wild and uneducated guess: Plesk Obsidian 18.0.37 Update 1 was released today. Maybe the release of the new version coincided with your automated update and maybe some mirrors and/or release files were not fully populated/updated yet, resulting in inconsistent package information.

But then again that's pure speculation...

 

[Bitpalast]

Indeed the last step of the removal procedure of duplicate packages
# plesk installer update --repatch
has upgraded the 18.0.37 to MU #1. But even in this case, an auto-upgrade should not break in the first place. If such a change in files on the upgrade servers causes an interruption, the upgrade itself should not be possible while these files are upgraded on the Plesk repositories. It is an unsatisfying situation, because it takes a lot of time to fix. I also wonder why in such cases the installer routine cannot fix this itself by running the repair steps. After all, repair was possible with no other changes to the underlying system or files, so installer should have been able to fix the situation itself.