poor qmail performance

[bartu]

Hi,
i've been expiriencing lately major problem with qmail speed on centralized mail server. It accepts incoming emails (you have to wait a long time for it) and places them in the queue. Problem is, that email are delivered to loacal recipients so slowly that queue keeps growing bigger. I haven't noticed such problem for outgoing emails.

Form observing top and vmastat i assumed that it's related to HDD work.
************
here's lsof output:
root@cmail:~# lsof |wc -l
3579
root@cmail:~# lsof |grep qmail|wc -l
586
************
and vmstat output:
root@cmail:~# vmstat 1 10
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 6 0 930480 99480 738756 0 0 368 155 119 136 3 1 74 22
0 6 0 930236 99488 738884 0 0 132 248 211 128 0 0 75 25
0 5 0 929440 99500 739144 0 0 260 468 305 469 2 0 75 23
0 4 0 929320 99504 739276 0 0 136 468 241 114 0 0 75 25
1 4 0 928892 99512 739404 0 0 124 348 232 275 1 0 72 27
0 2 0 928512 99520 739396 0 0 32 608 203 166 0 0 75 25
0 4 0 926696 99536 739788 0 0 176 336 359 486 7 1 68 24
2 2 0 924724 99576 740632 0 0 600 900 377 603 22 1 49 28
0 3 0 923584 99592 740616 0 0 12 516 203 492 8 2 50 39
0 4 0 922872 99592 740616 0 0 32 592 226 192 2 0 65 32
************
I've also made a test that showed that qmail is responsible for this
root@cmail:~# hdparm -t /dev/sda

/dev/sda:
Timing buffered disk reads: 20 MB in 3.32 seconds = 6.02 MB/sec
root@cmail:~# /etc/init.d/qmail stop
root@cmail:~# /etc/init.d/xinetd stop
Stopping internet superserver: xinetd.
root@cmail:~# killall -9 qmail-smtpd
root@cmail:~# hdparm -t /dev/sda

/dev/sda:
Timing buffered disk reads: 366 MB in 3.00 seconds = 122.00 MB/sec
************
I can add that CPU is used mosly on processess like: syslog, spamd, mysql, qmail processes

 

[stupidnic]

What do your mail logs say it is working on?

Looking at your vmstat it looks like you are waiting on IO to some degree, but nothing that should cause this level of problem.

I am specifically interested in any entries in the mail log file that reference your current concurrency settings, as well as any errors.

Apr  3 07:58:39 plesk1 qmail: 1207227519.874865 status: local 0/10 remote 0/20

 

[bartu]

Hi,

i've upgraded raid controller firmware and it helped.

concurrency is set to that:
/var/qmail/bin/qmail-showctl|grep concurrency
silent concurrency limit: 1000.
concurrencylocal: (Default.) Local concurrency is 10.
concurrencyremote: (Default.) Remote concurrency is 20.

there are no errors in system log.
I:'ve also noticed that in plesk web interface qmail is sometimes inactive as a system process for a moment. That happens when system is under havy load.

I plan recompileing kernel, and chnaging some raid settings. Maybe this will help

 

[stupidnic]

What type of RAID card are you using? What underlying drive type (SATA2, SCSI 320, etc)? what RAID level? Also, what kernel version, and what distro? Also what type of filesystem? Also what type of CPU (core counts, dual proc, etc)?

Double check the local concurrency in your logs to make sure you are not hitting the 10/10 limit under heavy load. For busy servers you can generally go to local 20 and remote 40 if you really need to.

I have seen some issues with apic on certain versions of motherboards causing some timeout issues on some hard drive controllers, so I am just trying to get a clearer picture of your hardware.

 

[bartu]

I have 3 sata disks at raid5, CPU: 2x Xeon 3 GHz, kernel is now 2.6.24.4 OS is Ubuntu 6.06.2. Filesystem is EXT3.
Right now situation is stable, but sill under heavy load (which shouldn;t be hevy for this machine) queue is growing. Usually in those situations it gets unloaded after few minutes.
I get the impression that server is running as it sholud