Plesk's milter for Postfix crashes on a malformed email

[burnleyvic]

Plesk 10.4.4#MU20 on CentOS 5.7 64bit
I have this funny email which crashes Plesk's milter. The email is attached, is named 3.new.gz and to reproduce the issue one must send it to an email address hosted on a Plesk server running Postfix with milter.
Important: the bug doesn't replicate on 9.5.4 when using the old smtp integration with /usr/lib/plesk-9.0/postfix-queue

When Plesk crashes the SMTP client gets the old good "451 4.3.0 Error: queue file write error" SMTP message after Data. In Plesk server's maillog I see these entries:
postfix/cleanup[26056]: 32E55237004E: message-id=<[email protected]>
postfix/cleanup[26056]: warning: incoming/32E55237004E: too many reverse jump records
postfix/cleanup[26056]: warning: cleanup_find_header_end: read file incoming/32E55237004E: Success
postfix/cleanup[26056]: panic: cleanup_milter_error: missing errno to error flag mapping
postfix/master[11415]: warning: process /usr/libexec/postfix/cleanup pid 26056 killed by signal 6
When this happens the message is copied in /var/spool/postfix/incoming queue and postfix tries to requeue it later, which will generate another crash and another similar email queued, leading to a constant growth of queued messages. See the attachment named postfix_mailqueue-day.png

If I disable Plesk's milter and use clamav with clamav-milter I get:
postfix/cleanup[26541]: 79AF11089821C: milter-reject: END-OF-MESSAGE from mx1.example.net.au[x.x.x.x]: 4.7.1 Service unavailable - try again later; from=<sender@address> to=<recipient@address> proto=ESMTP helo=<mx1.example.net.au>

To reproduce the crash, download 3.new.gz attachment and send it like this:
gunzip 3.new.gz | /var/qmail/bin/qmail-inject -f sender@address recipient@address
where recipient@address is hosted on a Plesk 10.4.4 running Postfix as MTA with psa-pc-remote

I'll try to obtain a core dump in Postfix, perhaps it will help.

 

[burnleyvic]

A quick follow-up from Parallels support:

------------------------------------------------------
The problem has been reproduced by product maintenance team, but root cause has not yet been found. Further in-depth investigation is needed by Parallels Plesk development team [...]
At this moment, we do not have ETA on availability of the fix for this issue and the only workaround is to switch to QMail in order to avoid this problem from happening.
------------------------------------------------------

Other workarounds include:
- scrap Plesk's milter and use another non-vulnerable product in place;
- call another milter before Plesk that will reject/refuse this type of emails. This is what we've opted for;
- use Postfix advanced configuration capabilities to block these emails before they're sent to the milter.

This Plesk vulnerability can turn into a major issue if no workaround is being applied. With a carefully crafted attack one can easily disable most, if not all the hosting services on a vulnerable installation by filling up the disk space, not to mention the CPU being used full-throttle by cleanup(8) process when such an email makes its way to the 'incoming' queue.
Because on the seriousness of this problem we expect Parallels to publish in 48 hours a notification advising its customers on the issue and the available workaround(s), otherwise both Parallels and their customers (us included) will be at risk after having this vulnerability disclosed on high traffic mailing-lists and websites specialized in such things, such as http://seclists.org/fulldisclosure/ and http://secunia.com/community/advisories/report_vulnerability/

 

[dash]

Thanks for report!
We've investigated the issue and found solution.
Attached is tar.gz archive with fixed binaries for cents5, 6 and debian 6
The fix will be delivered with nearest microupdate


1. /etc/init.d/pc-remote stop
2. extract archive and place correspondent to your OS into /usr/lib[64]/plesk-9.0/
3. /etc/init.d/pc-remote start

 

[burnleyvic]

Great news, the crash is definitely not there anymore! I'm testing the patch right now on one of the affected servers running CentOS 5 64bit and I don't get cleanup process to sigabrt anymore. There is an associated log entry like this one for each failure:
"Mar 19 10:05:48 host06 /usr/lib64/plesk-9.0/psa-pc-remote[18179]: Header's limit is exceeded(2000)... Spam?"
which is then followed by:
"4.7.1 Service unavailable - try again later;"
Before considering the issue fixed, one more question please: In "Header's limit is exceeded(2000)" message, what does 2000 value mean? Is this the length in bytes of one header? Or the length of a header line? How can this restriction affect from now on legitimate emails that include a large number of recipients, for example the
newsletters? We want to have this fixed, but not by enforcing arbitrary hardcoded limits that could disrupt businesses.

 

[dash]

Great news, the crash is definitely not there anymore! I'm testing the patch right now on one of the affected servers running CentOS 5 64bit and I don't get cleanup process to sigabrt anymore.

Really good news! Thanks for your feedback.

Before considering the issue fixed, one more question please: In "Header's limit is exceeded(2000)" message, what does 2000 value mean?

The limit means "maximum amount of identical headers per email".
Identical headers means header names (not including values), like a lot of "Content-type:" in email you provided.
We believe 2000 is quite reasonable limitation for real, not malformed emails.

 

[burnleyvic]

(The attachment part of this post was removed at Parallels' kind request)
dash, thank you very much for the explanations. This means the fix is specifically targeting one case, corrupted emails with many identical headers. But it doesn't deal with variants of this email. A modified version of this eml is bypassing your current restriction and crashes Postfix's cleanup(8) process the same way, in which case you'll need to work on a more generic fix. You should already have all the details to replicate the issue.
While we're at this, *I think* Parallels developers are in a good position to get in touch with Postfix developers and perhaps submit a bug against Postfix, looks like all it takes to crash Postfix is a custom email and a milter that apparently corupts that email before copying it into Postfix incoming queue.

 

[dash]

We are working on the solution.

 

[dash]

Plesk microupdate 10.4.4 MU#23(with the fix included) has been released.
Please check.
http://kb.parallels.com/113530

 

[fearz]

Anyone confirm this?

Anyone confirm this?

 

[Sven76]

After the last microupdate#23 of Plesk we still habe the problem that e-mails received with hours of delay. In the mail log, we have the following entries.

i replaced the real sender an recipient address with [email protected] (Sender) and [email protected] (recipient)

Here i see, we receive the mail ~3 hours later...


Mar 28 16:36:53 mail postfix/smtpd[25814]: C12752324: milter-reject: DATA from mailserver1.other.de[XXX.XX.XX.XXX]: 451 4.7.1 Service unavailable - try again later; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<mailserver1.other.de>
Mar 28 17:07:02 mail postfix/smtpd[27402]: 1061E2324: milter-reject: DATA from mailserver1.other.de[XXX.XX.XX.XXX]: 451 4.7.1 Service unavailable - try again later; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<mailserver1.other.de>
Mar 28 17:52:52 mail postfix/smtpd[28630]: 5359A2324: milter-reject: DATA from mailserver1.other.de[XXX.XX.XX.XXX]: 451 4.7.1 Service unavailable - try again later; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<mailserver1.other.de>
Mar 28 18:22:54 mail postfix/smtpd[29487]: 2EF842324: milter-reject: DATA from mailserver1.other.de[XXX.XX.XX.XXX]: 451 4.7.1 Service unavailable - try again later; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<mailserver1.other.de>
Mar 28 19:22:59 mail postfix/qmgr[7963]: 519A92324: from=<[email protected]>, size=210347, nrcpt=4 (queue active)

------------------
And here one detail entry:
Mar 28 17:07:02 mail postfix/smtpd[27402]: connect from mailserver1.other.de[XXX.XX.XX.XX]
Mar 28 17:07:02 mail postfix/smtpd[27402]: 1061E2324: client=mailserver1.other.de[XXX.XX.XX.XX]
Mar 28 17:07:02 mail greylisting filter[27407]: Starting greylisting filter...
Mar 28 17:07:02 mail greylisting filter[27407]: Timeout finished
Mar 28 17:07:02 mail /usr/lib64/plesk-9.0/psa-pc-remote[29839]: handlers_stderr: SKIP
Mar 28 17:07:02 mail /usr/lib64/plesk-9.0/psa-pc-remote[29839]: SKIP during call 'grey' handler
Mar 28 17:07:02 mail greylisting filter[27408]: Starting greylisting filter...
Mar 28 17:07:02 mail /usr/lib64/plesk-9.0/psa-pc-remote[29839]: handlers_stderr: DEFER
Mar 28 17:07:02 mail /usr/lib64/plesk-9.0/psa-pc-remote[29839]: DEFER during call 'grey' handler
Mar 28 17:07:02 mail /usr/lib64/plesk-9.0/psa-pc-remote[29839]: Message aborted.
Mar 28 17:07:02 mail postfix/smtpd[27402]: 1061E2324: milter-reject: DATA from mailserver1.other.de[XXX.XX.XX.XX]: 451 4.7.1 Service unavailable - try again later; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<mailserver1.other.de>
Mar 28 17:07:02 mail postfix/smtpd[27402]: disconnect from mailserver1.other.de[XXX.XX.XX.XX]

Best regards
Sven

 

[dash]

After the last microupdate#23 of Plesk we still habe the problem that e-mails received with hours of delay. In the mail log, we have the following entries.

Why do you think this is the same problem with malformed emails from initial post?

 

[Sven76]

Because i read the log entry

postfix/cleanup[26541]: 79AF11089821C: milter-reject: END-OF-MESSAGE from mx1.example.net.au[x.x.x.x]: 4.7.1 Service unavailable - try again later; from=<sender@address> to=<recipient@address> proto=ESMTP helo=<mx1.example.net.au>

and we also have the message:

4.7.1 Service unavailable - try again later; from=<sender@address> to=<recipient@address> proto=ESMTP ...

ok, I would have time to read it properly.....

Sorry, i will open a new Thread....