500 Error after automatic micro-update - Plesk 10.4

[KirkM]

Plesk CP is dead with a 500 server error after it auto-updated last night. I have tried everything my limited knowledge came up with and enlisted the help of my server customer support with no luck. I have been searching these forums and the web for an answer with no luck. Here is the last update from my server support:

The command in the server that is erroring out is:

/usr/bin/sw-engine-cgi -c /usr/local/psa/admin/conf/php.ini -d auto_prepend_file=auth.php3 -u psaadm

If you run the command as root it works. If you run the command as psaadm, the Plesk user, then it fails. I also tried running it as sw-cp-server. Here are the errors that occurred with both Plesk users:


bash-3.2$ /usr/bin/sw-engine-cgi -c /usr/local/psa/admin/conf/php.ini -d auto_prepend_file=auth.php3
Could not open Repository at "/etc/sw/keys": Cannot open file
Status: 500 PleskFatalException
Expires: Fri, 28 May 1999 00:00:00 GMT
Last-Modified: Mon, 05 Dec 2011 01:57:18 GMT
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Pragma: no-cache
P3P: CP="NON COR CURa ADMa OUR NOR UNI COM NAV STA"
Content-type: text/plain; charset=utf-8

ERROR: PleskFatalException
Unable to connect to database: get_admin_password() failed: file_get_contents() failed: Undefined index: PLESK_DEBUG_SQL

0: common_func.php3:150
psaerror(string 'Unable to connect to database: get_admin_password() failed: file_get_contents() failed: Undefined index: PLESK_DEBUG_SQL')
1: auth.php3:107
PleskFatalException: Unable to connect to database: get_admin_password() failed: file_get_contents() failed: Undefined index: PLESK_DEBUG_SQL
file: /usr/local/psa/admin/plib/common_func.php3
line: 150
code: 0
trace: #0 /usr/local/psa/admin/auto_prepend/auth.php3(107): psaerror('Unable to conne...')
#1 {main}

PHP Warning: Cannot modify header information - headers already sent by (output started at /usr/local/psa/admin/plib/PleskException.php:44) in /usr/local/psa/admin/plib/PleskException.php on line 27
ERROR: PleskFatalException
Unable to connect to database: get_admin_password() failed: file_get_contents() failed: Undefined index: PLESK_DEBUG_SQL

0: common_func.php3:150
psaerror(string 'Unable to connect to database: get_admin_password() failed: file_get_contents() failed: Undefined index: PLESK_DEBUG_SQL')
1: auth.php3:107
PleskFatalException: Unable to connect to database: get_admin_password() failed: file_get_contents() failed: Undefined index: PLESK_DEBUG_SQL
file: /usr/local/psa/admin/plib/common_func.php3
line: 150
code: 0
trace: #0 /usr/local/psa/admin/auto_prepend/auth.php3(107): psaerror('Unable to conne...')
#1 {main}

PHP Fatal error: Uncaught exception 'Zend_Exception' with message 'No entry is registered for key 'config'' in /usr/local/psa/admin/plib/Zend/Registry.php:147
Stack trace:
#0 /usr/local/psa/admin/plib/CommonPanel/Exception.php(63): Zend_Registry::get('config')
#1 /usr/local/psa/admin/plib/CommonPanel/Exception.php(43): CommonPanel_Exception::_sendRuntimeReportXML(Object(PleskFatalException), 1)
#2 /usr/local/psa/admin/plib/PleskException.php(49): CommonPanel_Exception::sendNotification(Object(PleskFatalException))
#3 /usr/local/psa/admin/plib/PleskException.php(10): report_crash('Unable to conne...', Array, 'PleskFatalExcep...', 500, Object(PleskFatalException))
#4 [internal function]: plesk_exception_handler(Object(PleskFatalException))
#5 {main}
thrown in /usr/local/psa/admin/plib/Zend/Registry.php on line 147
bash-3.2$ /usr/bin/sw-engine-cgi -c /usr/local/psa/admin/conf/php.ini -d auto_prepend_file=auth.php3 -u psaadm
initgroups() failed: Operation not permitted

 

[IgorG]

Did you try to fix your Plesk with bootstrapper repair instead by running some commands as part of complex upgrade process?

 

[KirkM]

Yes, when I ran bootstrapper, it messed up the entire server. Mail stopped working and all sites went to the Apache default page plus the 500 server error on the CP was still there.

My server host support got the sites back up (not sure what he did) and I got the mail working by following his suggestion of using autoinstaller to switch to Qmail from Postfix and then back to rebuild the config files.

Now we are back to where we started after the MU#5 broke the Plesk CP.

Any suggestions? I really need to do some work with things in the CP and phpMyAdmin and am stuck until this gets solved. My server hosts are pretty stumped and want to disable SELinux to see if that gives a clue if things work without it enabled. I am not too keen on that, but don't have any other options at this point. It is over my head.

 

[KirkM]

I disabled SELinux and ran bootstrapper again successfully. Unfortunately, Plesk CP is still broken.

This is very frustrating. It's not like I messed around and broke something. I just woke up this morning to a broken CP because of a bad auto-update.

Is there any way to rebuild plesk or somehow debug this problem caused by MU#5?

 

[IgorG]

Do you have any related errors in /var/log/sw-cp-server/error_log ?

 

[KirkM]

Endless amounts of this:

2011-12-04 10:12:47: (connections.c.299) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure
2011-12-04 10:12:47: (connections.c.299) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure
2011-12-04 10:12:47: (connections.c.299) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure
2011-12-04 10:12:47: (connections.c.299) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure
2011-12-04 10:12:47: (connections.c.299) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure
2011-12-04 10:12:47: (connections.c.299) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure
2011-12-04 10:12:47: (connections.c.299) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure
2011-12-04 10:12:47: (connections.c.299) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure
2011-12-04 10:12:47: (connections.c.299) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure
2011-12-04 10:12:47: (connections.c.299) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure
2011-12-04 11:05:52: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 0 load: 133
2011-12-04 11:05:55: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2011-12-04 11:07:20: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 0 load: 139
2011-12-04 11:07:20: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 1 load: 139
2011-12-04 11:07:23: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2011-12-04 11:07:23: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2011-12-04 11:08:48: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 0 load: 139
2011-12-04 11:08:48: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 1 load: 139
2011-12-04 11:08:48: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 2 load: 139
2011-12-04 11:08:51: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2011-12-04 11:08:51: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2011-12-04 11:08:51: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2011-12-04 11:10:18: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 0 load: 136
2011-12-04 11:10:18: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 1 load: 136
2011-12-04 11:10:18: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 2 load: 136
2011-12-04 11:10:18: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 3 load: 136
2011-12-04 11:10:21: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2011-12-04 11:10:21: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2011-12-04 11:10:21: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2011-12-04 11:10:21: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2011-12-04 11:11:47: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 0 load: 136
2011-12-04 11:11:47: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 1 load: 136
2011-12-04 11:11:47: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 2 load: 136
2011-12-04 11:11:47: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 3 load: 136
2011-12-04 11:11:47: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 4 load: 136
2011-12-04 11:11:50: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2011-12-04 11:11:50: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2011-12-04 11:11:50: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2011-12-04 11:11:50: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2011-12-04 11:11:50: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock

 

[IgorG]

http://kb.parallels.com/en/8119

 

[KirkM]

Didn't work. I don't think it is a brute force attack. That error stopped many hours ago. The error that keeps coming up is this one:

2011-12-05 02:16:15: (mod_fastcgi.c.1000) the fastcgi-backend /usr/bin/sw-engine-cgi -c /usr/local/psa/admin/conf/php.ini -d auto_prepend_file=auth.php3 -u psaadm failed to start:
2011-12-05 02:16:15: (mod_fastcgi.c.1004) child exited with status 1 /usr/bin/sw-engine-cgi -c /usr/local/psa/admin/conf/php.ini -d auto_prepend_file=auth.php3 -u psaadm
2011-12-05 02:16:15: (mod_fastcgi.c.1007) if you try do run PHP as FastCGI backend make sure you use the FastCGI enabled version.
You can find out if it is the right one by executing 'php -v' and it should display '(cgi-fcgi)' in the output, NOT (cgi) NOR (cli)
For more information check http://www.lighttpd.net/documentation/fastcgi.html#preparing-php-as-a-fastcgi-program
2011-12-05 02:16:15: (mod_fastcgi.c.1012) If this is PHP on Gentoo add fastcgi to the USE flags
2011-12-05 02:16:15: (mod_fastcgi.c.1105) [ERROR]: spawning fcgi failed.
Must be suid root
2011-12-05 02:16:17: (mod_fastcgi.c.1000) the fastcgi-backend /usr/bin/sw-engine-cgi -c /usr/local/psa/admin/conf/php.ini -d auto_prepend_file=auth.php3 -u psaadm failed to start:
2011-12-05 02:16:17: (mod_fastcgi.c.1004) child exited with status 1 /usr/bin/sw-engine-cgi -c /usr/local/psa/admin/conf/php.ini -d auto_prepend_file=auth.php3 -u psaadm
2011-12-05 02:16:17: (mod_fastcgi.c.1007) if you try do run PHP as FastCGI backend make sure you use the FastCGI enabled version.
You can find out if it is the right one by executing 'php -v' and it should display '(cgi-fcgi)' in the output, NOT (cgi) NOR (cli)
For more information check http://www.lighttpd.net/documentation/fastcgi.html#preparing-php-as-a-fastcgi-program
2011-12-05 02:16:17: (mod_fastcgi.c.1012) If this is PHP on Gentoo add fastcgi to the USE flags
2011-12-05 02:16:17: (mod_fastcgi.c.1105) [ERROR]: spawning fcgi failed.
Must be suid root
2011-12-05 02:16:18: (mod_fastcgi.c.1000) the fastcgi-backend /usr/bin/sw-engine-cgi -c /usr/local/psa/admin/conf/php.ini -d auto_prepend_file=auth.php3 -u psaadm failed to start:
2011-12-05 02:16:18: (mod_fastcgi.c.1004) child exited with status 1 /usr/bin/sw-engine-cgi -c /usr/local/psa/admin/conf/php.ini -d auto_prepend_file=auth.php3 -u psaadm
2011-12-05 02:16:18: (mod_fastcgi.c.1007) if you try do run PHP as FastCGI backend make sure you use the FastCGI enabled version.
You can find out if it is the right one by executing 'php -v' and it should display '(cgi-fcgi)' in the output, NOT (cgi) NOR (cli)
For more information check http://www.lighttpd.net/documentation/fastcgi.html#preparing-php-as-a-fastcgi-program
2011-12-05 02:16:18: (mod_fastcgi.c.1012) If this is PHP on Gentoo add fastcgi to the USE flags
2011-12-05 02:16:18: (mod_fastcgi.c.1105) [ERROR]: spawning fcgi failed.


From my earlier post:

The command in the server that is erroring out is:

/usr/bin/sw-engine-cgi -c /usr/local/psa/admin/conf/php.ini -d auto_prepend_file=auth.php3 -u psaadm

If you run the command as root it works. If you run the command as psaadm, the Plesk user, then it fails. I also tried running it as sw-cp-server and it failed too.

 

[KirkM]

running php -v shows this, which the log says is wrong:

[root@secure ~]# php -v
PHP Deprecated: Directive 'safe_mode' is deprecated in PHP 5.3 and greater in Unknown on line 0
PHP 5.3.8 (cli) (built: Oct 31 2011 18:26:52)
Copyright (c) 1997-2011 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2011 Zend Technologies
with the ionCube PHP Loader v4.0.10, Copyright (c) 2002-2011, by ionCube Ltd.

Or do I not understand this?

 

[KirkM]

After going around in circles with my server hosting company support for 24 hours it is finally fixed. As I originally proposed, evidently on deaf ears over there, it was corrupt files, almost certainly caused by a blown automatic micro-update. Parallels solved the problem. Here is the solution in this case if anyone else has this issue:

_____________________________

Parallels support updated that it is resolved:
-----
I am so glad to inform you that the issue with Plesk Panel showing 500 - Internal Server Error is fixed.

On my deeper investigation, I found that some files related with the rpm packages of plesk 10.4.4 were corrupted.

Hence, I have reinstalled those packages and the issue is fixed. Please verify the below snippets and the screen-shots.

The reinstalled packages are as listed below:

--

sw-cp-server-1.0-7.201106221135.centos5.x86_64.rpm
sw-doctrine-1.0.4-0.280352.noarch.rpm

Thanks,

--

Shonima V N
Technical Support Engineer
Parallels

Knowledge base: http://kb.parallels.com

--------

[root@secure support_backup]# rpm -Uvh sw-doctrine-1.0.4-0.280352.noarch.rpm
Preparing... ########################################### [100%]
1:sw-doctrine ########################################### [100%]
[root@secure support_backup]# rpm -Uvh sw-cp-server-1.0-7.201106221135.centos5.x86_64.rpm
Preparing... ########################################### [100%]
groupadd: group sw-cp-server exists
1:sw-cp-server ########################################### [100%]
Starting SWsoft control panels server... already running. [ OK ]
[root@secure support_backup]#

--------

 

[FergusB]

Really Useful

Really useful thank you - how do you reinstall the packages in the manner you describe?

 

[KirkM]

Run the two commands listed at the bottom of the post (assuming you are running centos5.x86_64):

# rpm -Uvh sw-doctrine-1.0.4-0.280352.noarch.rpm
# rpm -Uvh sw-cp-server-1.0-7.201106221135.centos5.x86_64.rpm

 

[FergusB]

Still no luck...

Thanks,
I've tried that and I'm getting the message that both those packages are already installed

Do you have any ideas how else to resolve this or how I would reinstall those packages, assuming that they are, as you say corrupt. Pretty much to the letter it seems like I'm facing the same problem as you did.

Best,
Fergus

 

[KirkM]

As must be obvious from my posts, I am no Linux expert, quite the contrary. My research shows that running either one of these should work, but I TAKE NO RESPONSIBILITY FOR THE OUTCOME. Perhaps someone else here with more knowledge can confirm the safety and effectiveness of these commands when reinstalling rpm packages.

The first is to simply force it to reinstall. I don't know what effect this has on possible dependencies, whether they will be added as in a normal install (I would guess they would, but don't know for sure):

rpm -Uvh --force sw-doctrine-1.0.4-0.280352.noarch.rpm
rpm -Uvh --force sw-cp-server-1.0-7.201106221135.centos5.x86_64.rpm

The other way is to use the --oldpackage switch so the server will allow the reinstall under the guise of it being a downgrade (even though it isn't). It just tells the system that it is ok to reinstall this already installed package even if it is the same or older version of the one currently installed:

rpm -Uvh --oldpackage sw-doctrine-1.0.4-0.280352.noarch.rpm
rpm -Uvh --oldpackage sw-cp-server-1.0-7.201106221135.centos5.x86_64.rpm

Again, I GIVE NO ASSURANCES THAT THESE WILL WORK FOR YOU OR WILL NOT DO HARM. My research shows they should work to reinstall those packages. I sincerely hope this helps since I know how much is sucks to have your server messed up when you are trying to do business. Best of Luck to you.

 

[FergusB]

Thank you very much KirkM, I forced the latter of those two the rpm -Uvh --oldpackage sw-cp-server-1.0-7.201106221135.centos5.x86_64.rpm and that seems to have brought it back up!

 

[KirkM]

That's great! I have had my share of server hangs over the years and I know how awful it is to be in that spot and what a relief it is to get it fixed. I am glad you are back on track.

Cheers

 

[Tozz]

Although possible, I doubt very much this is a bug in the microupdates. As far as I know (I'm personally using Debian, which uses .deb's instead of .rpm's) systems using .RPM's verify the downloaded RPM using MD5 and CRC checksum. Actually downloading a corrupt file would be noticed by the package manager.

So, if you still have any corrupted files (which you had), I would personally look at the integrity of your system. It might have bad RAM, you could have a corrupted filesystem or perhaps you have faulty hardware which is corrupting data.

 

[KirkM]

No, I don't believe the actual rpm was corrupt. I believe files do not get installed properly and end up corrupted during the auto update.

The Plesk update feature has been shaky at best since I have been using the panel. I don't know why, but it seems like it can burp during an update at any time and mess up the installation. I would call it "fragile", for lack of a better description.

After 10 years, 5 servers (with different hardware and config on each), and more of these situations than I care to remember , the consistent factor has been the Plesk auto update program. This forum is littered with examples from others as well. It is better now than it was back in the 9.x days. It actually didn't work at all back then. But it is still a gamble every time it runs.

One time, a couple of years ago, I had a server get dinged by an auto update and used the situation to update to new hardware with a complete clean install. I did no modifications at all and sure enough, I ran the Plesk updater and it broke the box. After my server provider got into it and then escalated to Plesk, it was ... wait for it ... FILES THAT *SOMEHOW* GOT CORRUPTED DURING THE PLESK UPDATE. Maddening? Yes. Shocking? No.

The Plesk panel has its strengths and weaknesses. The updater is a big weakness.