smtp auth not working - 421 out of memory

[pixeljunkie]

We have a Vserver from server4you.de with min 512MB / max 1GB RAM.
It took me a few nights to find out why smtp-auth width Plesk 8.01 installed doesn`t work.

It seems, that the auth-process gets kicked because of a memory limit.

I checked with telnet and used a CRAM-MD5 maker to send the right password string and i get a
"421 out of memory".

What the hell?

---clip--------------------
220 xxxxx.de ESMTP
EHLO client
250-xxxxx.de
250-AUTH=LOGIN CRAM-MD5 PLAIN
250-AUTH LOGIN CRAM-MD5 PLAIN
250-STARTTLS
250-PIPELINING
250 8BITMIME
AUTH CRAM-MD5
334 PDI0NTE4LjExNTY4NDE2M333ZXNvbm8tNS5kZT4=
c3ZlbkBwaXhlbGp1bmtpZS5kZSA3OGU2MGQ3NDQxNmMyNzZjY2FhNDNmZTdiYjllZjk4NQ==
421 out of memory (#4.3.0)
535 auth failure
---clap--------------------

I checked the available RAM with a vmemusage script provided by the hoster and it shows 312MB used of 512 available. So that should not be the problem.

I found this about increasing the "softlimit":
http://www.gossamer-threads.com/lists/qmail/users/130727

and this:
http://www.linux.com/howtos/Qmail-ClamAV-HOWTO/x277.shtml

But that didn`t help me, because i can`t find the line or even the right script where i should add/change the softlimit.

Any ideas?