Plesk Panel - error - backend died

[MislavO]

Hello, I've noticed since 4pm today that I'm running following problem with my Plesk Panel:
- plesk panel itself is not accessible (maybe from time to time) - I'm getting 500 - Internal Server Error

I've found that:
# /var/log/sw-cp-server/error_log

is full of following errors:

2013-01-17 19:10:18: (mod_fastcgi.c.2830) fcgi-server re-enabled: 0 /opt/psa/tmp/sw-engine.sock
2013-01-17 19:16:02: (mod_fastcgi.c.1746) connect failed: Connection refused on unix:/opt/psa/tmp/sw-engine.sock-0
2013-01-17 19:16:02: (mod_fastcgi.c.3072) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 4

I've found on forum that this could be brute-force attack on the panel, but as described here:
http://kb.parallels.com/en/8119

but there is no such error in logs. Also, i'm not able to find any /var/log/secure file on my server, it doesn't exist. This file is mentioned here:
http://forum.parallels.com/showthread.php?t=208198

Load on server is normal, like 1-2 max. Memory is also fine.

I've also tried:
http://forum.parallels.com/showthread.php?t=112676
http://forum.parallels.com/showthread.php?t=112880
- changing apc.shm_size and other apc values has no effect
- /usr/bin/sw-engine-cgi -c /usr/local/psa/admin/conf/php.ini -d auto_prepend_file=auth.php3 -u psaadm (no errors)

Running Plesk 11.0.9, MU #31.
Debian GNU/Linux 6.0.6 (squeeze)

I can't also do a upgrade, i get email and when i click on the link this is shown:

Execution failed.
Command: autoinstaller
Arguments: Array
(
[0] => --select-product-id
[1] => plesk
[2] => --select-release-current
[3] => --upgrade-installed-components
)

Details:
BUSY: Update operation was locked by another update process.

But that is not my problem atm. MU #31 is already active few weeks and this problem occurred TODAY.

Please note: ALL WEBSITES on the server are working just fine, they load in less then a second, so this
problem I'm facing is only related to plesk panel it self.

 

[IgorG]

BTW, instead of /var/log/secure Debian uses /var/log/auth.log

 

[MislavO]

Yeah....Meanwhile, i figure that out by googling and i didn't find any symptoms like "ssl handshake" in logs, but interesting is that i have 500 records in 10-15min, i'm not sure is this normal or not.

Jan 18 07:07:01 axion CRON[25638]: pam_unix(cron:session): session closed for user <user>
Jan 18 07:07:01 axion CRON[25644]: pam_unix(cron:session): session opened for user <user>
Jan 18 07:07:01 axion CRON[25642]: pam_unix(cron:session): session closed for user <user>
Jan 18 07:07:01 axion CRON[25636]: pam_unix(cron:session): session closed for user <user>
Jan 18 07:07:01 axion CRON[25645]: pam_unix(cron:session): session closed for user <user>
Jan 18 07:07:01 axion CRON[25644]: pam_unix(cron:session): session closed for user <user>

Records like that...(talking about /var/log/auth.log )

 

[IgorG]

Have you seen this Error - 500 hub? http://kb.parallels.com/114882

 

[MislavO]

Thanks for the link, articles:

http://kb.parallels.com/en/114058
- there is no PPSB or WPB in my Plesk (not using them - disabled)

http://kb.parallels.com/en/9755
- i can access Licence Management
# sysctl -w kernel.yama.ptrace_scope=0
error: "kernel.yama.ptrace_scope" is an unknown key

http://kb.parallels.com/en/1571
- there is no tomcat app

http://kb.parallels.com/en/114382
- no such errors in logs

http://kb.parallels.com/113418
http://kb.parallels.com/113466
- no such errors in logs - i've tried to set min-procs already before to 1, but no affect, I've found that here:
http://forum.parallels.com/showthread.php?t=112676

http://kb.parallels.com/8683
- backup files are not downloading

http://kb.parallels.com/en/5977
- I'm running Linux with MySQL

 

[MislavO]

# /var/log/auth.log

Jan 18 15:28:33 <host> su[2690]: pam_unix(su:session): session opened for user popuser by (uid=0)
Jan 18 15:28:34 <host> su[2690]: pam_unix(su:session): session closed for user popuser
Jan 18 15:28:34 <host> su[2705]: Successful su for popuser by root
Jan 18 15:28:34 <host> su[2705]: + ??? rootopuser
Jan 18 15:28:34 <host> su[2705]: pam_unix(su:session): session opened for user popuser by (uid=0)
Jan 18 15:28:35 <host> su[2705]: pam_unix(su:session): session closed for user popuser
Jan 18 15:28:35 <host> su[2716]: Successful su for popuser by root
Jan 18 15:28:35 <host> su[2716]: + ??? rootopuser
Jan 18 15:28:35 <host> su[2716]: pam_unix(su:session): session opened for user popuser by (uid=0)
Jan 18 15:28:35 <host> su[2716]: pam_unix(su:session): session closed for user popuser
Jan 18 15:28:35 <host> su[2744]: Successful su for popuser by root
Jan 18 15:28:35 <host> su[2744]: + ??? rootopuser
Jan 18 15:28:35 <host> su[2744]: pam_unix(su:session): session opened for user popuser by (uid=0)

EVERY second is this in logs....

 

[MislavO]

Problem is now fixed, Parallels Support solved my problem.

Problem was with "sw-cp-server" package. They tried to manually reinstall it, but they were getting errror:

Due to some reasons installer wasn't able kill the process correctly causing this fault. In order to resolve the issue, I killed existing 'sw-cp-serverd' process manually and installed 'sw-cp-server' package using above mentioned command.

Plesk is now up and running!

 

[HarryBleckert]

I got the very same problem after auto-upgrading to 11.0.9 Update #36. Installed OS is Ubuntu 10.04.4 LTS.

How did Parallels Support fix the problem ???

 

[MislavO]

As i quoted above, they reinstalled the package.

Whole quote from email:

It was not possible to start sw-cp-server (Plesk webserver).

--
root@<HOSTNAME> ~ # /etc/init.d/sw-cp-server start
Starting SWsoft control panels server... stale pidfile. failed
--

I tried performing manual reinstallation of 'sw-cp-server' package but it reported following errors;

--
root@<HOSTNAME> ~ # dpkg -i sw-cp-server_1.0-8.201205141804.debian60_amd64.deb.1
(Reading database ... 100040 files and directories currently installed.)
Preparing to replace sw-cp-server 1.0-8.201205141804.debian60 (using sw-cp-server_1.0-8.201205141804.debian60_amd64.deb.1) ...
Stoppping SWsoft control panels server... stale pidfile. ok
Unpacking replacement sw-cp-server ...
Setting up sw-cp-server (1.0-8.201205141804.debian60) ...
Starting SWsoft control panels server...failed
dpkg: error processing sw-cp-server (--install):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
--

Due to some reasons installer wasn't able kill the process correctly causing this fault. In order to resolve the issue, I killed existing 'sw-cp-serverd' process manually and installed 'sw-cp-server' package using above mentioned command.

Just incase i suggest you to contact them.

 

[IgorG]

I got the very same problem after auto-upgrading to 11.0.9 Update #36. Installed OS is Ubuntu 10.04.4 LTS.

How did Parallels Support fix the problem ???

I can say only that MU#36 has absolutely nothing concerning sw-cp-server. Therefore upgrade to MU#36 can't be a reason of problems with sw-cp-server. As I see in your support ticket #1589560 that problem under support investigation now.

 

[HarryBleckert]

Plesk support was very responsive and helpful. After extensive log reviews they figured the problem was cause by bruteforce attacks.
I could not verify bruteforce attacks after reviewing logs again. And fail2ban plus firewall did not show alarming numbers. Apache was always responsive, server load was low.

However, in between log reviews, the problem was gone, panel became responsive again and sw-engine-cgi stopped to open hundreds of instaces and eating RAM (but no CPU usage).

I am still nervous because I don´t know what caused the problem.

 

[IgorG]

BTW, This issue with sw-engine-cgi has been fixed in MU#38 - http://kb.parallels.com/115579