Issue [17.5.3] Plesk for Linux patches, still a bloody disgrace

[burnley]

Folks, so you've just decided to push #29 which, according to Change Log for Plesk contains a new feature shipped together with the latest Roundcube 1.2.7 packages to fix a zero day vulnerability. Great, good on youse! The problem is, actually are, at least 2:
- Why on Earth do you keep shipping new features mixed with security patches?
- Why do you keep insisting with this 1995-ish approach of reconfiguring & restarting services when it is NOT necessary? This Roundcube release DOES NOT REQUIRE YOU TO STOP APACHE AND THUS TAKE ALL THE SITES DOWN, does it? Has anyone @plesk looked into what the latest Roundcube package update consists in? I'll tell you: it only updates files, that's all! When will any of the hundreds of Plesk engineers realize that you don't need to stop Apache and reconfigure the server + all the webmail vhosts just to overwrite some PHP files?
You've still got a bloody long way in learning how to package your stuff properly. One should only run "rpm -q --scripts plesk-roundcube" to realize how disastrous your packaging is.
Will you ever release a product that's properly packaged? No, you don't have to answer this.

P.S. Guess putting up the license fees up 300% will help you fund the development of properly packaged Plesk panel, won't it?

 

[burnley]

... and here's the icing on the cake: this morning I've just patched one test 17.5.3 instance, bringing up to #MU30, for yet another plesk-roundcube-1.2.7 package. Same problem, of course: reconfigure the webserver's server config and the webmail vhosts, even if it's not necessary.
And btw, why do you push patches without changelogs? Not only your packaging is flat broken, but you're taking your paying customers by surprise!

 

[Igor Borisov]

Updates with security fixes and features:

The update you’ve mentioned doesn’t have any new features (for reference, new features are marked by [+], not by [*], as per the Changelog legend), so we’re not entirely sure what exactly we keep doing. Are you talking about the small non-security fix/improvement included in the aforementioned update? If so, such fixes/improvements are often critical for someone else and delaying them just because we’re releasing an update with a security fix wouldn’t be reasonable. Moreover, if a security Plesk update has something else besides the security fixes, we don’t see it as a big deal, because there’s no way to choose which Plesk updates should be installed - any update will require installing all the previous ones anyway. If you have a particular case where a Plesk security update also delivered a new feature and this created a problem for you, please let us know, so we can check it.


Roundcube packaging:

Unfortunately, it’s not so simple. The Roundcube package contains Apache configuration file inside. To be sure that the Apache configuration from this file is applied after the update regardless of any manual configuration changes made by customers, we have to restart the Apache. We cannot use graceful reload because, well, based on the collective experience of our engineers gathered after fixing thousands of servers, graceful reload can lead to Apache service failure. Of course, if we had hundreds of engineers, we could’ve probably gathered even more information on this, but I think we’ve seen more than enough evidence during the almost-20-year-long history of Plesk that graceful reload causes Apache service failure too often for our liking. Nevertheless, we understand your pain, so we’ll review the Roundcube package and see if we can reduce the number of Apache restarts if possible. Meanwhile, I can advise you to choose minimum load time for performing upgrades on production servers.


Updates without changelogs:

As you can imagine, it takes some time to publish a Plesk update for all OSes. We update Release Notes only after we have verified that everything was uploaded correctly and that the update can be successfully downloaded and installed. This creates a brief window of time where the update is technically available, but the Changelog isn’t published yet (because we’re still veryfing that everything is good to go before doing the announcement). We’re sorry to hear that your discovery of another Plesk update happened during this brief window, and we’ll see if we can make this window shorter in the future.

Let us know if there’s something else we at Plesk R&D can help you with. Thanks!

 

[Tomek]

As you can imagine, it takes some time to publish a Plesk update for all OSes.

It makes me a little worried.

Is plesk is safe solution? Or there is nothing to worry about? How does Plesk handle this, compared to other panels? (I do not hating, seriously asking)

 

[burnley]

Igor, thanks for replying, much appreciated. You've made few valid points there. See inline my answers:

Are you talking about the small non-security fix/improvement included in the aforementioned update? If so, such fixes/improvements are often critical for someone else and delaying them just because we’re releasing an update with a security fix wouldn’t be reasonable.

Yes, that's what I was referring to. You've pushed a Roundcube security update, fair enough, you had to.

Moreover, if a security Plesk update has something else besides the security fixes, we don’t see it as a big deal, because there’s no way to choose which Plesk updates should be installed - any update will require installing all the previous ones anyway.

This is one of the problems I've been battling with when it comes to Plesk updates sinde day dot. Because of the way you choose to push updates, you're sometimes causing too much unnecessary downtime.

Roundcube packaging:
Unfortunately it’s not so simple. The Roundcube package contains Apache configuration file inside. To be sure that the Apache configuration from this file is applied after the update regardless of any manual configuration changes made by customers, we have to restart the Apache.

Two points:
- Have you looked in detail into this Roundcube update? It was simply code change to close that security hole. No web server reconfiguration was needed *at all*.
- When upgrading Roundcube Plesk reconfigures all the domains webmail configuration. Before or during the reconfiguration (don't know exactly when because Plesk doesn't log its update operations properly) Apache service was actually stopped. A bit heavy handed, don't you think? On a Plesk server with 700+ virtual hosts the reconfiguration takes quite a bit of time and it's completely unacceptable to cause long server-wide downtime to patch few php files.

We cannot use graceful reload because, well, is based on the collective experience of our engineers gathered after fixing thousands of servers, graceful reload can lead to Apache service failure. Of course, if we had hundreds of engineers, we could’ve probably gathered even more information on this, but I think we’ve seen more than enough evidence during the almost-20-year-long history of Plesk that graceful reload causes Apache service failure too often for our liking.

Yes, Apache's graceful reload has always been a pita and still is in 2.4. With of without Plesk we've always had issues with the unreliability of graceful reload.

Nevertheless, we understand your pain, so we’ll review the Roundcube package and see if we can reduce the number of Apache restarts if possible. Meanwhile I can advise you to choose minimum load time for performing upgrades on production servers.

Simplifying the packaging and the the delivery of updates is the first thing you can look into. If an operation isn't necessary, avoid it. Especially if that operation is causing service downtime on busy servers. If it's necessary only run it *once*. Specifically to Roundcube: this is the epitome of a simple package. A bunch of core, plugins and config files. The package upgrade should be trivial. Looking at the files list:

rpm -ql plesk-roundcube | grep -v "\/usr\/share\/psa-roundcube"
/etc/httpd/conf/plesk.conf.d/roundcube.htaccess.inc
/etc/psa-webmail/roundcube
/etc/psa-webmail/roundcube/roundcube.conf
/usr/local/psa/etc/logrotate.d/roundcube
/usr/share/doc/plesk-roundcube-1.2.7
/usr/share/doc/plesk-roundcube-1.2.7/CHANGELOG
/usr/share/doc/plesk-roundcube-1.2.7/INSTALL
/usr/share/doc/plesk-roundcube-1.2.7/LICENSE
/usr/share/doc/plesk-roundcube-1.2.7/README.md
/usr/share/doc/plesk-roundcube-1.2.7/UPGRADING
/var/log/plesk-roundcube
/var/tmp/plesk-roundcube

Which one of the above files would require a server configuration rebuild? Besides, compare the outputs between these 2 commands:
rpm -q --scripts plesk-roundcube | grep -v "^$\|^#" | wc -l
6730
rpm -q --scripts roundcubemail | wc -l
8
Where roundcubemail is the RPM packaged by Remi. 6730 lines for the Plesk package? What for, to manipulating some PHP and Apache files? I understand you need to cater for various distros and releases, but seriously, it's too much. Most of that code pre/post/prein/postun code is simply not required for plesk-roundcube. Try to keep it simple

 

[burnley]

On the Apache restart, this is what I see on one of the servers affected:
/var/log/plesk/install/autoinstaller3.log
[...]
[2017-11-15 08:48:51.246502] Bootstrapper has finished action (exec time: 1 sec.): parent_name='panel', sequence='post', stage='execute', sequence_order='0', operation='install', exec_cmd='/usr/local/psa/bootstrapper/pp17.5.3-bootstrapper/bootstrapper.sh post-install psa-vhost'', m_arch='', output: ~empty
[2017-11-15 08:48:52.489325] Bootstrapper has finished action (exec time: 1 sec.): parent_name='roundcube', sequence='post', stage='execute', sequence_order='0', operation='install', exec_cmd='/usr/local/psa/bootstrapper/pp17.5.3-bootstrapper/bootstrapper.sh post-install roundcube'', m_arch='', output: ~empty
[2017-11-15 08:48:59.724952] ===> Cumulative APS controller upgrade (final stage) has been started.
[2017-11-15 08:49:47.925453] Bootstrapper has finished action (exec time: 0 sec.): parent_name='PLESK_17_5_3', sequence='post', stage='execute', sequence_order='9998', operation='install', exec_cmd='test ! -x /usr/local/psa/admin/sbin/packagemng || /usr/local/psa/admin/sbin/packagemng --set-dirty-flag'', m_arch='', output: ~empty
/var/log/httpd/error_log-20171119
[...]
[Wed Nov 15 08:49:49.554258 2017] [mpm_prefork:notice] [pid 24523] AH00170: caught SIGWINCH, shutting down gracefully
[...]
2 seconds after bootstrapper logged its last entry Apache was gracefully reloaded. Unfortunately the reload was done before plesk started to reconfigure the server.

L.E. Correction, my mistake for misreading the logs. Apache hasn't been reloaded, but shut down as part of the roundcube upgrade. Soon after Plesk started rebuilding the server httpd configuration. This behaviour is simply unacceptable considering the reason (package upgrade full stop) and the impact (all the sites are down for God knows how many minutes). Surely this is something you must look into.

 

[Sergey L]

It makes me a little worried.

Is plesk is safe solution? Or there is nothing to worry about? How does Plesk handle this, compared to other panels? (I do not hating, seriously asking)

You don't have to be worried in this context. Plesk needs to check that each and every user can access update before announcing it. If you were able to get it, you are fine.
We are working on reducing gap between publishing and announcement, but anyway most people will discover update only next day so small gap has little practical impact

 

[Igor Borisov]

Thank you for your feedback and sorry for making you wait an answer for a long time,

- Have you looked in detail into this Roundcube update? It was simply code change to close that security hole. No web server reconfiguration was needed *at all*.

As I've already said, we'll review Roundcube package to get rid of unnecessary restarts. Sorry for inconvenience, but please be patient, it could take some time.

6730 lines for the Plesk package? What for, to manipulating some PHP and Apache files? I understand you need to cater for various distros and releases, but seriously, it's too much. Most of that code pre/post/prein/postun code is simply not required for plesk-roundcube. Try to keep it simple

We're building our packages using unified build system to deliver them fast and make sure all packages are seamlessly integrated with installed Plesk ecosystem. It needs more steps when installing or upgrading package than in generic Remi package. Not all functions are required by every particular package, but having to support individual script set for every single package may significantly slower updates and new packages delivery.

 

[trialotto]

@burnley, @Sergey L and @Igor Borisov

The interesting part about this whole discussion is that everyone is contributing with valid concerns and responses to that concerns.

This is good and solid feedback from @burnley, enabling Plesk Staff to improve the Plesk product.

In my humble opinion, the entire discussion still misses an important factor: the time between vulnerability discovery and the patch introduced in Plesk.

In fact, security patches are often not a problem in a well-balanced server setup and the delivery time of security patches for Plesk is acceptable, but not prioritized.

Again in my humble opinion, @burnley pinpoints a bottleneck that also applies to security patches.

In this topic thread, many alternative viewpoints have been discussed.

But I have not read an alternative that allows a split between

1 - upgrades and patches that require the "full process of restarting services", (and)
2 - upgrades and patches that are minor of nature OR updates and patches that need to be prioritized, such as security related patches and updates

and the upgrades and patches from point 2 can be run daily, whereas the upgrades and patches from point 1 can run less frequent, hence also allowing Plesk Team more time to create a rock solid solution that is stable across all the supported OSes.

It is just a suggestion, in order to speed up the development of relevant components of Plesk.

Regards......