failed upgrade results in inconsistent DB and unfunctional panel

[Shantal]

Hi guys,

After last night's attempt to upgrade one of our old plesk servers (10.0.1), on RHEL 5.5, we are left with unfunctional control panel.

Following the KB http://kb.odin.com/en/113604, we see:

1) no /usr/local/psa/version file exists currently.

2) mysql -uadmin -p`cat /etc/psa/.psa.shadow` -D psa -e 'select * from upgrade_history order by upgrade_date desc limit 1'
+---------------------+--------------------------------+------------+
| upgrade_date | version_info | db_version |
+---------------------+--------------------------------+------------+
| 2010-12-01 10:52:30 | 10.0.1 RedHat el5 109101029.17 | 01090 |
+---------------------+--------------------------------+------------+

as you see the plesk was OLD. No line for the last night's upgrade, though in here.

3) but: rpm -q psa
psa-12.0.18-rhel5.build1200140606.15

So, the installation procedure finished partially, installing net packages, but not really updating all info.

Comparison of the MySQL scheme for DB psa with another server with the new installation shows that some minimal differences in some tables and a couple of missing tables. So, it looks like it _almost_ succeeded.

Comparison of the RPMs installed on the two servers, shows some differences in installed packages and versions. For example, there are 2 versions of psa-backup-manager and 2 versions of psa-migration-manager on the "malfunctioning" machine.

Further runs of the autoinstaller does not help, since it seems to think that everything is fully installed/updated.

We are running out of ideas how to proceed from this point. Control panel does not work...

We've looked trought the installation log files and Wonder if there is some log file that we are missing - where shold we look for messages in case of a failed installation?

Does anybody has any idea for how to improve the situation from now on. Any pointers will be helpful - we are at the end of our rope here.

Thanks for reading this far,
- Shantal

 

[UFHH01]

Did you try the bootstrapper - repair?

/usr/local/psa/bootstrapper/pp12.0.18-bootstrapper/bootstrapper.sh repair​

 

[Shantal]

Hi and thank you for your reply.

Yes, we tried that and things got a whole lot worse. The repair ran for some time and ended with errors (some SQL grant query fails).

Now all sites are down and in /var/log/sw-cp-server/error_log, we get:

2014/10/25 18:10:05 [crit] 23535#0: *1328 connect() to unix:/var/run/sw-engine.sock failed (2: No such file or directory) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: , request: "GET /login_up.php3 HTTP/ 1.1", upstream: "fastcgi://unix:/var/run/sw-engine.sock:", host: "name.domain.com:8443"
...
2014/10/25 18:12:41 [emerg] 11750#0: eventfd() failed (38: Function not implemented)

Any ideas?

 

[UFHH01]

For a quick info, you might as well use port 8443 to login to the Plesk panel.

Second, please use the command:

/usr/local/psa/admin/bin/autoinstaller --select-product-id plesk --select-release-current --reinstall-patch --install-component base​

... to make sure that all Plesk related software and depending packages are installed ( as well all patches ) and please post the logs from the installer and the bootstrapper and finally as well from apache ( and niginx if installed ?!? ) and if domains are not reachable, from the the domain - apache - nginx - error - logs as well, if you still experience problems ( Parallels Plesk Panel for Linux services logs and configuration files ( KB: 111283 ) ). It is much easier to investigate issues/problems with logs than with sentences like "it got worse" ( please... that isn't meant as an offence! ) The more corresponding logs you provide, the better might be the suggestions to help you with issues/problems.

 

[Shantal]

On the first point: at this moment, logging in to plesk control panel on 8443 is useless. After providing credentials, we are directly forwarded to the license management page, where it states that:
"
Error: The license key is invalid. In order to use Plesk, please obtain and install a new valid license key.
You are trying to use license key created for another version of Parallels Plesk. Contact Parallels sales representatives to obtain the proper license key.
The amount of currently used resources overrides the limits defined by your license.
The number of websites hosted on the server exceeds the limits defined by your license. You have 15 sites hosted; your license allows hosting only 0 sites.
...
"
(we have a valid license, but for some reason plesks decides now that it's invalid - since the upgrade last night)

So, currently plesk control panel is not helping.

Second: I'll get back to you on that one after we've run it - thank you for your advice, we do appreciate every piece of info.

 

[Shantal]

Hi again

We run this command:
/usr/local/psa/admin/bin/autoinstaller --select-product-id plesk --select-release-current --reinstall-patch --install-component base

It went fine. Please find log file attached (autoinstaller3.txt).

Current status is unchanged:
- domain websites are working fine
- we can't use control panel: we get the same complain about license right after login
- when we click "retrieve keys" from the page where the license error is shown, it we get the error:

ERROR: SB_Facade_Exception_Generic
Undefined property "isPublished" in object "SB_Site".

- the last record in psa.upgrade_history table still shows
# mysql -uadmin -p`cat /etc/psa/.psa.shadow` -D psa -e 'select * from upgrade_history order by upgrade_date desc limit 1'
+---------------------+--------------------------------+------------+
| upgrade_date | version_info | db_version |
+---------------------+--------------------------------+------------+
| 2010-12-01 10:52:30 | 10.0.1 RedHat el5 109101029.17 | 01090 |
+---------------------+--------------------------------+------------+
- there is no version file: /usr/local/psa/version
- nginx is probably not installed: rpm -qa | grep nginx gives no output (not sure what the rpm name is for nginx)
- please find attached /var/log/sw-cp-server/error_log

Here is some extra information about boostrap repair that was run earlier:
- the command failed on the first run giving this error:

ERROR while trying to execute SQL query, the query was: SET SESSION sql_mode=REPLACE(@@SESSION.sql_mode, 'NO_AUTO_CREATE_USER', '');
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON horde.*
TO 'horde'@'localhost' IDENTIFIED BY '.....';
FLUSH PRIVILEGES;

When running the above manually it was clear that it was failing on "FLUSH PRIVILEGES" caused by missing table mysql.servers. Then we run mysql_upgrade which fixed that problem. On the next run of bootstrap repair, it finished fine but while running it was showing errors typically like this one:

Fatal Error:
Could not instantiate PDO. PDOException: SQLSTATE[HY000] [1045] Access denied for user 'horde'@'localhost' (using password: YES)
In /usr/share/psa-pear/pear/php/Horde/Db/Adapter/Pdo/Base.php on line 48​

Please find plesk_12.0.18_repair.txt attached.

Any ideas how to proceed from here?

Help is highly appreciated.

 

[UFHH01]

The Horde - error is not really relevant right now, you should consider removing Horde, just to be sure that all other upgrades/updates/patches are o.k. and you can then concentrate on the license issue.
If you never used the Webpresence builder, you might want to uninstall this component right now, just to be sure as well, that all other feature work as expected ( see why I suggest this after the autoinstaller suggestions ). I would not recommend removing this component, if you have current domains working with it.

First we have a look at the current status of all components:

/usr/local/psa/admin/bin/autoinstaller --select-product-id plesk --select-release-current --show-components --show-options​

You now see the components and the options for them. With this list in mind, you might want to de- and re-install components, which are buggy right now and install as well components which might be missing. The commands would be:

/usr/local/psa/admin/bin/autoinstaller --select-product-id plesk --select-release-current --remove-component COMPONENT_NAME
You might remove more than one component, if you repeat the string "--remove-component COMPONENT_NAME" with the additional component you would like to remove​

/usr/local/psa/admin/bin/autoinstaller --select-product-id plesk --select-release-current --install-component COMPONENT_NAME
Choose the name of the component which you would like to install - it is possible to install several components in one command by repeating the string "--install-component COMPONENT_NAME"

or use

/usr/local/psa/admin/bin/autoinstaller --select-product-id plesk --select-release-current --install-everything
... to install all components​

/usr/local/psa/admin/bin/autoinstaller --select-product-id plesk --select-release-current --upgrade-installed-components
This command will upgrade your components.
​

If you experience failures with these commands, because of a license-key issue, you might choose the string "--ignore-key-errors".

If you would like the autoinstaller eMail - report, with a detailed protocoll, you can add the string "--notify-email [email protected]"


Now to your license issue:

The KB - article 112 453 shows, that the failure "SB_Facade_Exception_Generic" results of an enabled MySQL strict mode. You might want to fix this with the suggestions in the KB - article:

Error while accessing Parallels Web Presence Builder: "Base table or view not found" ( KB - article 112 453 )​

.. and let's see afterwards, if the retrieving key now works as expected.

 

[Shantal]

Thank you UFHH01 for the precise and fantasic answer!

We removed horde and followed kb112453 to recreate webpresence's db. Afterwards we could click "Retrieve Keys" and the license error disappeared. Now Plesk control Panel is mostly working. There is some strange issues with enabling domains and users. We'll open a support ticket for this.

I have a questions about the installed components on the system. There are a few shown in error state:

php53 [error] - PHP5.3 support
phpfpm [error] - php-fpm support for nginx (requires one of 'atomic' or 'remi' repositories enabled)

My guess is that those packages are not installed but why would they be shown as error? Are they required? Specially regarding php53 - can one switch per domain which version of php to be used (I mean if there are 2 version of php).

We highly appreciate your help!

 

[UFHH01]

Hi Shantal,

PHP itself is not installed by Plesk and it is not shipped with it, nor does Plesk upgrade any PHP - versions on your system.

Release Notes for Parallels Plesk 12.0 for Linux Systems ( please see: Software Requirements )
​

There is a fantastic PHP - version for your system ( Redhat ):

PHP Panda! Multi-PHP installer for Plesk​

and you will find all needed informations in this thread as well there.