Resolved Failed to update the Panel (due to autoinstall.plesk.com server problems)

[mow]

Moin,

yesterday plesk wanted to upgrade itself from 18.0.41 to 42.

Plesk Installer 3.42.0 (revision ee29a3f9022c) started at (timezone CET) Wed Mar 16 01:17:20 2022 Command line arguments: /var/cache/parallels_installer/parallels_installer_Debian_9.0_x86_64_3.42.0 upgrade plesk 18.0.42 --email *redacted*

Installer runs, downloads a bunch, adds temporary repositories, gets package lists, adds more repositories, gets pp18.0.42-bootstrapper sw-engine-cli-3.42, upgrades the database, then wants to update packages again:

Expand/Collapse Block: apt-get

sw-engine-cli-3.42 is already the newest version (3.42.1-debian.9.220228.1507).
pp18.0.42-bootstrapper is already the newest version (18.0-v.debian.9+p18.0.42.1+t220314.1713).
The following package was automatically installed and is no longer required:
php7.0-xsl
Use 'apt autoremove' to remove it.
Suggested packages:
python3-apt-dbg python-apt-doc
The following NEW packages will be installed:
plesk-python3-regex plesk-rrd python3-apt python3-yaml The following packages will be upgraded:
libaps plesk-backup-utilities plesk-completion plesk-control-panel
plesk-core plesk-core-utilities plesk-distupgrade plesk-dns-bind-driver
plesk-dovecot plesk-dovecot-core plesk-dovecot-imap-driver
plesk-dovecot-pigeonhole plesk-engine plesk-l10n plesk-librdbmspp
plesk-lmlib plesk-mail-pc-driver plesk-platform-runtime plesk-release
plesk-repair-kit plesk-roundcube plesk-service-node-utilities
plesk-task-manager plesk-ui-library plesk-web-hosting postfix psa
psa-autoinstaller psa-firewall psa-locale-base-en-us psa-logrotate
psa-mail-driver-common psa-phpmyadmin sw-collectd sw-engine sw-nginx
36 upgraded, 4 newly installed, 0 to remove and 14 not upgraded.
Need to get 127 MB of archives.
After this operation, 2805 kB of additional disk space will be used.

Then it started to get those packages, but after a few, autoinstall.plesk.com failed randomly:

Expand/Collapse Block: download failures

Get:16 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-rrd amd64 1.7.2-v.debian.9+p18.0.42.0+t220208.1413 [582 kB]
Err:17 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 sw-collectd amd64 5.11.0-v.debian.9+p18.0.42.0+t220215.1558
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
Err:18 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-completion amd64 18.0-v.debian.9+p18.0.42.0+t220208.1413
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
Get:19 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-service-node-utilities amd64 18.0-v.debian.9+p18.0.43.0+t220228.1947 [3488 kB]
Get:20 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-backup-utilities amd64 18.0-v.debian.9+p18.0.42.1+t220314.1713 [3383 kB]
Err:21 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-task-manager amd64 18.0-v.debian.9+p18.0.42.0+t220208.1413
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
Err:22 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 psa-phpmyadmin all 5.1.3-v.debian.9+p18.0.42.0+t220214.1144
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
Get:23 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 psa-locale-base-en-us all 18.0-v.debian.9+p18.0.42.1+t220314.1713 [335 kB]
Get:24 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-core amd64 18.0-v.debian.9+p18.0.42.1+t220314.1713 [29.7 MB]
Ign:25 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 postfix amd64 3.5.14-v.debian.9+p18.0.42.0+t220208.1413
Get:26 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 psa-mail-driver-common amd64 18.0-v.debian.9+p18.0.43.0+t220228.1947 [1185 kB]
Err:27 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-mail-pc-driver amd64 18.0-v.debian.9+p18.0.42.0+t220208.1413
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
Get:28 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-distupgrade amd64 18.0-v.debian.9+p18.0.42.1+t220314.1713 [18.7 kB]
Err:29 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-dns-bind-driver amd64 18.0-v.debian.9+p18.0.42.0+t220221.1425
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
Err:30 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-dovecot-imap-driver amd64 18.0-v.debian.9+p18.0.42.0+t220208.1413
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
Ign:31 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-dovecot-pigeonhole amd64 0.5.18-v.debian.9+p18.0.42.0+t220208.1413
Ign:32 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-dovecot amd64 2.3.18-v.debian.9+p18.0.42.0+t220208.1413
Ign:33 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-dovecot-core amd64 2.3.18-v.debian.9+p18.0.42.0+t220208.1413
Get:34 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-l10n all 18.0-v.debian.9+p18.0.42.1+t220314.1713 [7772 kB]
Get:35 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-repair-kit amd64 18.0-v.debian.9+p18.0.42.1+t220314.1713 [3741 kB]
Get:36 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-web-hosting amd64 18.0-v.debian.9+p18.0.42.1+t220314.1713 [74.1 kB]
Ign:37 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 plesk-roundcube all 1.4.13-v.debian.9+p18.0.42.0+t220208.1413
Err:38 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 psa-autoinstaller amd64 3.42.0-debian.9.220218.1308
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
Err:39 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 psa-firewall amd64 18.0-v.debian.9+p18.0.42.0+t220208.1413
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
Ign:40 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/all amd64 sw-nginx amd64 1.20.2-v.debian.9+p18.0.42.0+t220208.1413
Err:25 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/extras amd64 postfix amd64 3.5.14-v.debian.9+p18.0.42.0+t220208.1413
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
Err:31 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/extras amd64 plesk-dovecot-pigeonhole amd64 0.5.18-v.debian.9+p18.0.42.0+t220208.1413
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
Err:32 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/extras amd64 plesk-dovecot amd64 2.3.18-v.debian.9+p18.0.42.0+t220208.1413
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
Err:33 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/extras amd64 plesk-dovecot-core amd64 2.3.18-v.debian.9+p18.0.42.0+t220208.1413
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
Err:37 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/extras amd64 plesk-roundcube all 1.4.13-v.debian.9+p18.0.42.0+t220208.1413
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
Err:40 http://autoinstall.plesk.com/pool/PSA_18.0.42_8472 stretch/extras amd64 sw-nginx amd64 1.20.2-v.debian.9+p18.0.42.0+t220208.1413
504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80] Fetched 76.7 MB in 1min 50s (692 kB/s)
E: Failed to fetch http://autoinstall.plesk.com/pool/PSA_18.0.42_8472/dist-deb-Debian-9.0-x86_64/base/sw-collectd_5.11.0-v.debian.9+p18.0.42.0+t220215.1558_amd64.deb 504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
E: Failed to fetch http://autoinstall.plesk.com/pool/PSA_18.0.42_8472/dist-deb-Debian-9.0-x86_64/base/plesk-completion_18.0-v.debian.9+p18.0.42.0+t220208.1413_amd64.deb 504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
(list of other failed downloads)
E: Failed to fetch http://autoinstall.plesk.com/pool/PSA_18.0.42_8472/extras-deb-Debian-9.0-x86_64/opt/hosting/nginx/sw-nginx_1.20.2-v.debian.9+p18.0.42.0+t220208.1413_amd64.deb 504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Launching component and product checkers...
Downloading file report-update: 0%
Downloading file report-update: 100% was finished.
Downloading file pool/PSA_18.0.42_8472/examiners/py_launcher.sh: 0% Downloading file pool/PSA_18.0.42_8472/examiners/py_launcher.sh: 100% was finished.
Error: An error occurred on attempt to install packages.
Attention! Your software might be inoperable.
Visit https://support.plesk.com/ to search for a solution.

Unfortunately searching for a solution was not successful because
https://support.plesk.com/hc/en-us/...-operations-until-Plesk-installation-is-fixed unfortunately 404s.

Since then, I got Package Update Manager notification mails complaining:
Some problems occurred with the System Updates tool on your server domain.com. Please resolve them manually.

Reason: 2022-03-16 06:25:27 INFO: pum is called with arguments: ['--list', '--repo-info', '--json']
2022-03-16 06:25:27 ERROR: Previous Plesk installer call finished unsuccessfully. PUM will not perform any update operations until Plesk installation is fixed.
2022-03-16 06:25:27 ERROR: Exited with returncode 50.

Thank you for making the upgrade process so fault tolerant for failures of your own server (504 Gateway Time-out means the request must have reached your server, so it's not a connectivity problem) and not including any hints about which commands to run to fix this.

Is plesk installer --select-release-current --reinstall-patch --upgrade-installed-components the correct command to try in this case, or if not, then what is?

 

[IgorG]

Err:37 Index of /pool/PSA_18.0.42_8472 stretch/extras amd64 plesk-roundcube all 1.4.13-v.debian.9+p18.0.42.0+t220208.1413 504 Gateway Time-out [IP: 2a02:6ea0:c000::6 80]

That means the server couldn't connect to autoinstall.plesk.com using its IPv6 address(2a02:6ea0:c000::6), which looks like a temporary network issue. The 504 Gateway timeout error could be caused by CDN issues when one of the servers becomes out of sync with the primary replica, and some packages fail to download.

You can verify the connectivity with the repository using the following command:

# curl -Ilk autoinstall.plesk.com:80

Also, using the next command, you can force the use of IPv4 for autoinstall.plesk.com in case there are any IPv6 routing issues on the service provider's side:

# echo "195.181.175.48 autoinstall.plesk.com" >> /etc/hosts

And proceed with the Plesk upgrade.

 

[mow]

That means the server couldn't connect to autoinstall.plesk.com using its IPv6 address(2a02:6ea0:c000::6), which looks like a temporary network issue. The 504 Gateway timeout error could be caused by CDN issues when one of the servers becomes out of sync with the primary replica, and some packages fail to download.

*nitpick* Our server could connect to autoinstall.plesk.com, it just wasn't able to answer properly. If it was a network issue, I wouldn't get a 504, but just a Connection timed out instead.

And proceed with the Plesk upgrade.

How? It doesn't restart on its own.

 

[IgorG]

How? It doesn't restart on its own.

As you wrote: plesk installer --select-release-current --reinstall-patch --upgrade-installed-components

 

[mow]

As you wrote: plesk installer --select-release-current --reinstall-patch --upgrade-installed-components

WARNING: Command line option '--reinstall-patch' is deprecated and will be ignored.
It runs, downloads a bunch of .inf files, synchronizes APT (with Hit:4 http://autoinstall.plesk.com/pool/PSA_18.0.41_8197 stretch InRelease!) and then says:
You already have the latest version of product(s) and all the selected components installed. Installation will not continue.
Same for plesk installer update.

pum is still stuck, /admin/pum/updates-list says Error: Unable to receive update information for the package(s)

 

[IgorG]

Try to perform upgrade to 18.0.42 with:

# plesk installer upgrade plesk 18.0.42

 

[mow]

(Oh, and /var/lock/parallels-panel-upgrade-failure.flag existed but was NOT mentioned in /admin/pum/updates-list like in Issue - docker.service failed)

# plesk installer upgrade plesk 18.0.42

Thanks, that worked.
Upgrade finished, the above lockfile is gone, pum shows updates again, and I could finally update the SSL packages.