Panel and qmail both very very slow

[rjshelq]

Plesk 11.0.9 is running exceptionally slow. This system had been happily running as expected for several months, but sometime between the morning of Nov 18 and the morning of Nov 19 something suddenly happened.

The major two symptoms are:

1) Emails can only be sent from a Perl script (invoking sendmail) at a rate of about one email every 7 to 8 seconds (on average).

2) Plesk login takes around 30 to 90 seconds, from the time I enter my username and password until the Plesk panel home screen is completely displayed. Changing from screen to screen in Panel often takes 30 seconds or more.

Everything seems to eventually work in the Plesk Panel (and Power panel), but everything takes about 5 times to 10 times longer than expected. I have not seen any outright errors on any Plesk screen, so the data seems to all be fine, but getting to it just takes a long long time.

Plesk and the email sending are the only things that seem to be running slowly. The web server (Apache) seems to be serving html and php web pages quickly, and without any obvious slow-down. I've run Perl test scripts to read and write to files a number of times (1000 times), and the file read/write times seem to be fine.

Here's what I've tried so far (roughly in the order in which they occurred)... none of which have had any useful effect on this problem whatsoever:

* rebooted server multiple times using Plesk Panel

* updated from 11.0.9 update #19 to update #25 using Plesk panel

* rebuilt qmail via ssh /usr/local/psa/admin/sbin/mchk --with-spam

* checked all databases via ssh mysqlcheck -uadmin -p`cat /etc/psa/.psa.shadow` -A

* ran bootstrapper via ssh /usr/local/psa/bootstrapper/pp11.0.9-bootstrapper/bootstrapper.sh

* changed from qmail to postfix using Plesk Panel

* changed MySQL to Plesk v5.5 using Plesk Panel

* rechecked all databases via ssh mysqlcheck -uadmin -p`cat /etc/psa/.psa.shadow` -A

* changed back from Postfix to qmail using Plesk Panel


None of those changes has had any useful effect on the slowness of either the email sending or Plesk Panel access.

Help.. please!

What could be wrong that would cause both Plesk Panel and qmail to be so dreadfully slow??

Any ideas of what to try next will be greatly appreciated.

 

[IgorG]

I'm impressed with the list of what you tried to do, but why you do not try to find something useful in the logs with help of KB article http://kb.parallels.com/111283?

 

[rjshelq]

Thank you for the reply Igor.

I did start off with studying the logs, and saw only one log (sw-cp-server/error_log) that had any complaints, but that log seemed to indicate a sporadic error, even though my problem is more or less continuous.

I looked for problems in these logs:

/var/log/sw-cp-server/error_log
/usr/local/psa/admin/logs/httpsd_access_log
/usr/local/psa/var/log/maillog
/var/log/messages
/var/log/mysqld.log
/usr/local/psa/admin/logs/panel.log
/var/lib/mysql/MyServer.net.err

But found nothing which seemed significant or repeating except for this sporadic error from /var/log/sw-cp-server/error_log:

2012-11-19 12:41:14: (mod_fastcgi.c.2746) FastCGI-stderr: PHP Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'ee48f5722c7f94da1c4a701f19df6b19' for key 'PRIMARY'' in /usr/local/psa/admin/plib/Zend/Db/Statement/Pdo.php:228
Stack trace:
#0 /usr/local/psa/admin/plib/Zend/Db/Statement/Pdo.php(228): PDOStatement->execute(Array)
#1 /usr/local/psa/admin/plib/Zend/Db/Statement.php(300): Zend_Db_Statement_Pdo->_execute(Array)
#2 /usr/local/psa/admin/plib/Zend/Db/Adapter/Abstract.php(479): Zend_Db_Statement->execute(Array)
#3 /usr/local/psa/admin/plib/Zend/Db/Adapter/Pdo/Abstract.php(238): Zend_Db_Adapter_Abstract->query('INSERT INTO `se...', Array)
#4 /usr/local/psa/admin/plib/Db/Adapter/Pdo/Mysql.php(19): Zend_Db_Adapter_Pdo_Abstract->query('INSERT INTO `se...', Array)
#5 /usr/local/psa/admin/plib/Zend/Db/Adapter/Abstract.php(575): Db_Adapter_Pdo_Mysql->query('INSERT INTO `se...', Array)
#6 /usr/local/psa/admin/plib/Zend/Db/Table/Abstract.php(1075): Zend_Db_Adapter_Abstract->insert('sessions', Array)
#7 /usr in /usr/local/psa/admin/plib/Zend/Db/Statement/Pdo.php on line 234

2012-11-19 13:11:20: (server.c.1543) server stopped by UID = 0 PID = 21846
......

2012-11-20 13:47:48: (mod_fastcgi.c.2746) FastCGI-stderr: PHP Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction' in /usr/local/psa/admin/plib/Zend/Db/Statement/Pdo.php:228
Stack trace:
#0 /usr/local/psa/admin/plib/Zend/Db/Statement/Pdo.php(228): PDOStatement->execute(Array)
#1 /usr/local/psa/admin/plib/Zend/Db/Statement.php(300): Zend_Db_Statement_Pdo->_execute(Array)
#2 /usr/local/psa/admin/plib/Zend/Db/Adapter/Abstract.php(479): Zend_Db_Statement->execute(Array)
#3 /usr/local/psa/admin/plib/Zend/Db/Adapter/Pdo/Abstract.php(238): Zend_Db_Adapter_Abstract->query('UPDATE `session...', Array)
#4 /usr/local/psa/admin/plib/Db/Adapter/Pdo/Mysql.php(19): Zend_Db_Adapter_Pdo_Abstract->query('UPDATE `session...', Array)
#5 /usr/local/psa/admin/plib/Zend/Db/Adapter/Abstract.php(634): Db_Adapter_Pdo_Mysql->query('UPDATE `session...', Array)
#6 /usr/local/psa/admin/plib/Zend/Db/Table/Abstract.php(1132): Zend_Db_Adapter_Abstract->update('sessions', Array, Array)
#7 /usr/local/psa/admin/plib/Ze in /usr/local/psa/admin/plib/Zend/Db/Statement/Pdo.php on line 234

2012-11-20 15:36:57: (server.c.1543) server stopped by UID = 0 PID = 9728
.....

However, since my problem is essentially continuous, those few errors did not seem to explain the problem (to me).

The mail log illustrates the slowdown, but provides no clue to me of how to fix it. Here's a sample of my maillog for sending just one email (using sendmail via ssh command line). Note that it took 19 seconds to finish sending one email. (the message body was only five words long)

There are two very long delays. Note that there is a 5 second delay after the line which says "SKIP during call 'check-quota' handler". And then a 10 second delay before the line that says "starting delivery 11"

What could cause these long delays??

Nov 19 20:57:46 MyServer qmail-queue-handlers[14286]: Handlers Filter before-queue for qmail started ...
Nov 19 20:57:49 MyServer qmail-queue-handlers[14286]: [email protected]
Nov 19 20:57:49 MyServer qmail-queue-handlers[14286]: [email protected]
Nov 19 20:57:49 MyServer qmail-queue-handlers[14286]: handlers_stderr: SKIP
Nov 19 20:57:49 MyServer qmail-queue-handlers[14286]: SKIP during call 'check-quota' handler
Nov 19 20:57:54 MyServer qmail-queue-handlers[14286]: starter: submitter[14310] exited normally
Nov 19 20:57:54 MyServer qmail: 1353387474.737528 new msg 47073695
Nov 19 20:57:54 MyServer qmail: 1353387474.737584 info msg 47073695: bytes 254 from <[email protected]> qp 14310 uid 0
Nov 19 20:58:04 MyServer qmail: 1353387484.342160 starting delivery 11: msg 47073695 to remote [email protected]
Nov 19 20:58:04 MyServer qmail: 1353387484.342244 status: local 0/10 remote 1/40
Nov 19 20:58:04 MyServer qmail-remote-handlers[14323]: Handlers Filter before-remote for qmail started ...
Nov 19 20:58:04 MyServer qmail-remote-handlers[14323]: [email protected]
Nov 19 20:58:04 MyServer qmail-remote-handlers[14323]: [email protected]
Nov 19 20:58:05 MyServer qmail: 1353387485.370246 delivery 11: success: 65.54.188.126_accepted_message./Remote_host_said:_250__<[email protected] t>_Queued_mail_for_delivery/
Nov 19 20:58:05 MyServer qmail: 1353387485.370356 status: local 0/10 remote 0/40
Nov 19 20:58:05 MyServer qmail: 1353387485.370407 end msg 47073695

The basic problem seems to be that Plesk, and anything that involves Plesk involvement (such as qmail) is running very, very slowly.

It takes as long as 90 seconds to log into the Plesk Panel! Yet when the data is finally shown on-screen, it is all correct.

It seems like the inter-module communication inside Plesk is very slow. Could it be related to your internal server? Or some software used for inter-process communication?

Help!

Any ideas or clues will be greatly appreciated.

 

[Faris Raouf]

I would be careful about putting 2 and 2 together and making 5 here.

Use top to see what the load is, and if something in particular is causing a high load. That single thing could be overloading the server in general and slowing things down. Yes, your web pages seem to be loading fast, but ignore that temporarily.

Remember that web pages might be cached, so will appear to load quickly when in fact the server is being slow.

The Plesk panel itself has no direct link to qmail's sending process as far as I remember. The messages will be in the qmail queue and will get sent even if the Plesk panel is not running. I don't think it has any link to mysql either for sending. This is why I'm kind of leaning towards there being some other thing causing a problem. If course if the plesk panel is causing the load then that might be the problem.

I know I'm not being very helpful in terms of pinpointing a cause - sorry :-(

 

[IgorG]

What about ps ax, top, free diskspace, hardware problems? Have you tried clean mail queue?

 

[rjshelq]

Yes, I have run utilities such as top, and ps aux. But they do not show anything which seems unusual to me.

The qmail queue has been cleaned and emptied. I have over 30 Gb of free disk space. Typical ram usage is around 650 MB out of 1 GB available.

My server provider (MediaTemple) has checked the server and claims there are no hardware problems. As a hardware test, I downloaded a large file (2 gb) from my web site, and it arrived in normal time. I have also downloaded smaller files via http such as mp3 and mp4 files, and they too arrive at normal speed. Files downloaded and/or uploaded via ssh also run at normal speed. I've written perl scripts to test disk read and write speed and they seem fine (typically around 38 msec to write 1000 short files).

So, the server seems fine, and anything that does not involve Plesk seems fine, but tasks such as getting into the Plesk Panel (or qmail's reliance on the use of Plesk handlers during emailing) are surprisingly slow.

I just logged into my Plesk Panel, and it took 92 seconds from the time I clicked on the "Log In" button until the Panel fully appeared on-screen. Then I clicked on "Manage in power User View" and it took 40 seconds to display that page. During all of that time, neither top nor ps aux showed anything extraordinary, sw-engine-cgi hit 5% cpu usage a few times, but everything else including mysql stayed even lower. Interesting, during most of that 132 seconds from log-in to display of the Power User View, there were seldom any processes using more than around 0.3% cpu and there were no memory hogs. The only thing I saw in the process list was that there were 4 seemingly identical sw-engine-cgi processes running, but perhaps that's normal: /usr/bin/sw-engine-cgi -c /usr/local/psa/admin/conf/php.ini -d auto_prepend_file=auth.php3 -u psaa

I believe that finding out what is taking 92 seconds to log-in and 40 seconds to get to Power User View will find the problem. So, let's just focus on that issue.

What could cause Plesk to take 92 seconds to simply log into my Plesk Panel?? and 40 seconds more to switch to "Manage in Power User View"?

 

[IgorG]

I think it would be more effectively if you contact Support Team. Experienced supporters will check this issue directly on your server and found a reason. It is really very difficult to do deep investigation in frame of forum discussion.

 

[rjshelq]

The hosting company eventually re-booted Virtuozzo, which solved this problem. The exact cause remains a mystery.

 

[IgorG]

The hosting company eventually re-booted Virtuozzo, which solved this problem. The exact cause remains a mystery.

Most probably it is related to lack of allowed resources for Plesk VPS. It should be investigated from Virtuozzo node side.