Issue Emails forwarded by Plesk fail DKIM checks with certain attachments

[Alex Presland]

Server operating system version

Ubuntu 22.04.3 LTS

Plesk version and microupdate number

Obsidian 18.0.57 Update #4

I'm running Plesk Obsidian 18.0.57 Update #4 on Ubuntu 22.04.3 LTS. This server was built in February 2023 from the AWS template. Under "Add and Remove Product Components" it shows that we're running Postfix [3.6.4-1ubuntu1-1] as the mail server, SpamAssassin [3.4.6-1ubuntu0.22.04.1], and Dovecot [2.3.21-v.ubuntu.22.04+p18.0.57.0+t231106.2014] for the IMAP/POP3 servers. The only extension which I think is relevant to email is Warden Anti-spam and Virus Protection [Version 4.00-1] by @danami .

I don't know whether this is a bug with Plesk, Warden, or a configuration issue, so I'm raising it here for discussion before submitting a bug report.

Some users have been complaining that some of their emails which are being forwarded through the server are sometimes being rejected by certain destinations. For the most part it works perfectly. Until now I've been unable to replicate this, but one affected user spotted that whenever they attached a DOCX or XLSX file it was rejected by certain destinations, but other attachments like PDF & JPG would go through without issue.

I have set up a test scenario and have successfully reproduced the issue and captured the email at various points on its journey. All emails were sent from "Microsoft Outlook for Microsoft 365 MSO (Version 2311 Build 16.0.17029.20028) 64-bit"

• EMAIL SOURCE: An email address outside the Plesk server coming from a domain with a DMARC record containing "p=reject"
• EMAIL ORIGINAL DESTINATION: An email address on the Plesk server with mailbox enabled and multiple forwarding destinations
• EMAIL FORWARDING DESTINATIONS (8 of them!): Another mailbox on the Plesk server, but under a different domain; Gmail; Yahoo; BT Internet; TalkTalk; iCloud; Hotmail; and a mailbox back on the source server)

TEST 1
Sending an HTML email (with no attachments) in to the Original Destination is successful and all forwarding recipients receive it.

TEST 2
Sending an HTML email (with a PDF attachment) in to the Original Destination is successful and all forwarding recipients receive it.

TEST 3
Sending an HTML email with a DOCX attachment in to the Original Destination is partially successful. It ends up in the Original Destination mailbox, the different domain mailbox on the same Plesk server, BT Internet, and the mailbox on the source server. It is rejected by Yahoo, TalkTalk, iCloud and Hotmail... all giving the reason that the DMARC validation failed. Where it made into a mailbox (instead of being rejected) it shows a DMARC fail.

I ssh-ed into the Plesk server and retrieved the email from /var/qmail/mailnames/[DOMAIN]/[LOCAL_PART]/Maildir/new/ for both the Original Destination and the different domain on the same Plesk server. Back on my PC I used WinMerge to compare them. I was shocked to find that the email forwarded within Plesk was failing DKIM & DMARC. Below are some snippets from the email headers explaining why for both "Original" and "Forwarded".

The first thing that I spotted was that a "Received" header had been line-wrapped:
Original

Received: from [12.34.56.78] (helo=DESKTOP3B)
by sending.server.name with esmtpsa (TLS1.2) tls TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
(Exim 4.96)
(envelope-from <[email protected]>)
id 1rFgTU-00Gg51-1D
for [email protected];
Tue, 19 Dec 2023 20:13:56 +0000

Forwarded:

Received: from [12.34.56.78] (helo=DESKTOP3B)
by sending.server.name with esmtpsa (TLS1.2) tls
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
(Exim 4.96)
(envelope-from <[email protected]>)
id 1rFgTU-00Gg51-1D
for [email protected];
Tue, 19 Dec 2023 20:13:56 +0000

Next were the Authentication-Results...
Original

Authentication-Results: destination.server;
dmarc=pass (p=REJECT sp=NONE) smtp.from=sending.domain header.from=sending.domain;
dkim=pass header.d=sending.domain;
spf=pass (sender IP is 23.45.67.89) smtp.mailfrom=[email protected] smtp.helo=sending.server.name

Forwarded (note the "dkim=fail" in red!)

Authentication-Results: destination.server;
dmarc=pass (p=REJECT sp=NONE) smtp.from=sending.domain header.from=sending.domain;
dkim=pass header.d=pubupdates.camra.org.uk;
dkim=fail header.d=sending.domain;
spf=pass (sender IP is 127.0.0.1) smtp.mailfrom=srs0=1nt2=h6=sending.domain=[email protected] smtp.helo=localhost

In the header of the forwarded email the reason for this is given (note the reason in red!):

Authentication-Results: destination.server (amavisd-new);
dkim=pass (1024-bit key) header.d=destination.domain
header.b="RZMWSriP"; dkim=fail (1024-bit key)
reason="fail (body has been altered)" header.d=sending.domain
header.b="i4IxasB+"

So now looking at the body portions of the email I spotted the line-wrapping of the "Content-Type" of the MS Word document:
Original

------=_NextPart_000_0267_01DA32B7.E5AFD240
Content-Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document;
name="Adding Account to Gmail_Rev1.0.docx"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename="Adding Account to Gmail_Rev1.0.docx"

Forwarded

------=_NextPart_000_0267_01DA32B7.E5AFD240
Content-Type:
application/vnd.openxmlformats-officedocument.wordprocessingml.document;
name="Adding Account to Gmail_Rev1.0.docx"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename="Adding Account to Gmail_Rev1.0.docx"

A PDF will have the following Content-Type... so won't be wrapped and won't fail DKIM / DMARC when forwarded.

Content-Type: application/pdf;

So... a number of questions:

1. Is there a configuration setting that I can change to fix this, or do people think that this is a bug which I should report formally?
2. If a configuration tweak, can this be the out-of-the-box behaviour
3. Is anyone able to reproduce this issue on their Plesk server?

Thanks in advance for your review and discussion.

 

[Peter Debik]

I'd omit the "by Plesk" in the headline, because this is a widely known issue with mail servers and forwarding on the internet. The problem is that recipients evaluate the sender of the mail, and that's the forwarder. I don't think there is a real fix for that.

 

[Alex Presland]

I'd omit the "by Plesk" in the headline, because this is a widely known issue with mail servers and forwarding on the internet.

I'm comfortable (and have proven & provided the evidence) that within my Plesk server the message body is being Changed / DKIM-corrupted through line wrapping.

 

[Peter Debik]

The same can be true for other configurations without Plesk, too.

 

[Alex Presland]

The same can be true for other configurations without Plesk, too.

Thanks Peter. In your like of work you are obviously familiar with email systems and say that this is common. Are you able to elaborate please? Do you know if this is commonly occurring in Postfix, SpamAssassin, Warden, etc. I get that Plesk is an integration of these many products, some of which are optional.

I'm pleased to report that a Debian server that I built for personal use (non-Plesk) and uses Exim4 and Spamassassin works perfectly in this scenario.

@danami Are you able to do any testing to prove that it isn't Warden causing this please?

 

[danami]

@Alex Presland You can uncheck the "SMTP milter" and "Non-SMTP milter" checkboxes under Settings -> Content Filter Settings -> Milter Settings to remove the amavisd-milter from Postfix for testing and see if DMARC will pass with it disabled.

 

[Alex Presland]

You can uncheck the "SMTP milter" and "Non-SMTP milter" checkboxes under Settings -> Content Filter Settings -> Milter Settings to remove the amavisd-milter from Postfix for testing and see if DMARC will pass with it disabled.

Thanks @danami. Tried that. Great idea, but sadly that doesn't fix the issue. Good that it rules out one more thing though - progress is being made!
Gmail shows this:

and

 

[Peter Debik]

Are you able to elaborate please?

What kind of explanation does a "it does not work" need? Sometimes it is a good idea to accept things as they are.

I see people trying to circumvent the mainstream solutions all the time, and of course they are failing. Maybe there is some special configuration for the special solution in Postfix. Maybe it can be done some way. I just don't want to waste time on these super-special-extra configurations that exist for some things. You may find something, then an update comes out and ... boom ... everything needs to be researched again, because the special configuration no longer works. So sorry, no, I would like to decline to discuss the technical details. It's not Plesk related anyway.

Maybe the root cause is the concept of forwarding everything instead of either mailing it to the right address to start with or to just store it in a mailbox and pull it from there.

 

[Kaspar]

Thanks @danami. Tried that. Great idea, but sadly that doesn't fix the issue. Good that it rules out one more thing though - progress is being made!
Gmail shows this:
View attachment 25108
and
View attachment 25109

@Alex Presland am I correct in seeing that with disabling to options suggested by @danami the DKIM validation no longer fails, while previously DKIM did fail?

 

[danami]

@Kaspar No he said that DMARC still failed with the amavisd-milter removed from the Postfix config.

 

[learning_curve]

I'm comfortable (and have proven & provided the evidence) that within my Plesk server the message body is being Changed / DKIM-corrupted through line wrapping.

Not seeking to cause any issues additional to those that you currently have / may have, but...

FWIW as we have quite similar Server OS setups (albeit we're using IONOS Cloud Servers but don't think you are / ALL of our DNS is managed external to Plesk but not sure about yours / We use individual SSL Certs for each hosted domain plus a SAN SSL Cert for the (Plesk) hosting domian (sub-domain), which by choice, includes all of the hosted domains too - This last conifg was orginally chosen for usage with Plesk (specifically for mail), but has since proven very useful elsewhere. Pretty sure you may not be doing the same thing) and... having run the same checks/tests as you have posted, we don't have any replication at all.

The big caveat being... that by choice and design, we do NOT use any Windows and/or Microsoft Hardware / Software anywhere at all
We have complete Linux / Ubuntu / macOS / iOS utilisation, so could only replicate your tests, within those parameters
Therefore... Certainly in our case c/w the caveat above, we're in agreement with @Peter Debik and his posts #2 and #4 in this thread
YMMV of course, but pretty sure it's not a problem exclusive to Plesk (or if it is, then it's a misconfig somewhere, not a design / operational fault)

 

[mow]

@Kaspar No he said that DMARC still failed with the amavisd-milter removed from the Postfix config.

Before it was DMARC pass DKIM fail, though.

@Alex Presland If you send from Outlook using an address on that server, are those headers wrapped too? (DKIM probably wouldn't fail in that case as the server generates the signature after mangling)

 

[danami]

Here is some info on the content-type header as to why these might be getting wrapped:

https://www.w3.org/Protocols/rfc1341/rfc1341.html

It states:
(4) Lines longer than 76 characters may be wrapped or truncated in some environments. Line wrapping and line truncation are STRONGLY DISCOURAGED, but unavoidable in some cases. Applications which require long lines should somehow differentiate between soft and hard line breaks. (A simple way to do this is to use the quoted-printable encoding.)

Unfortunately I wasn't able to replicate this as my email client uses "Content-Type: multipart/alternative;" when I attach .docx so nothing gets wrapped. If you check the Postfix config you should see that the amavisd-milter (inet:127.0.0.1:10024) runs before the Plesk milter. The Plesk milter (inet:127.0.0.1:12768) is the milter that signs the email.

# postconf smtpd_milters
smtpd_milters = inet:127.0.0.1:10024,inet:127.0.0.1:12768

 

[Alex Presland]

Thanks to all for your comments and constructive feedback. I've been busy and not able to reply properly. Hopefully I'll get time to reply after the next few days...

 

[Alex Presland]

Hi All. I got busy on my festive break and then got busy with paid work after that. Apologies for my lack of reply. The above is still an issue in my book, but very happy for events to supersede this.

I see that ARC was implemented in 18.0.58 with an issue still being in focus for some users:

Forwarded to devs - Arc errors in maillog

Username: TITLE Arc errors in maillog PRODUCT, VERSION, OPERATING SYSTEM, ARCHITECTURE Product version: Plesk Obsidian 18.0.58.0 OS version: AlmaLinux 8.9 x86_64 Build date: 2024/01/05 08:00 Revision: ec671a07e896ad1b354270e5a4c8597163abfc66 PROBLEM DESCRIPTION...

talk.plesk.com

My understanding is that working ARC authentication will solve the need for messages to be transited through a server unchanged.

 

[Bobbbb]

I'm also seeing issues with mail that is received by our servers running Plesk and Postfix, then forwarded to Gmail, for example. Certain (non-plaintext) messages (possibly containing 8bit content) fail DMARC and DKIM with "body hash did not verify".

I'd like to figure out if this is a Postfix issue, or if it is related to something the Plesk milter is doing.

I can work around the issue, for messages I personally send, by forcing all messages to use Quoted Printable in Thunderbird. Then the messages make it to the destination with valid DKIM. However, I can't change the mail client settings of all users who might send email to addresses on our servers.

 

[eLoneX]

Had a similar issue with "body hash did not verify" when forwarding emails to Hotmail/Gmail/Yahoo, it came down to Postfix line wrapping long HTML lines, basically is this:

"The line length limit for a 7-bit MIME part within an email, according to the MIME standard defined in RFC 2045 and the email format standard defined in RFC 5322, is 998 characters per line, not including the CRLF (Carriage Return, Line Feed) line ending characters. This limit is set to ensure compatibility across different mail transfer agents (MTAs) and email clients."

Since we couldn't change the incoming content, we've opted for adding this to main.cf:

smtp_line_length_limit = 0

Which fixed the problem, you can choose to have a bigger number instead of 0 (0 disables the limit), maybe that will help you too.

 

[Bobbbb]

Thanks. But in my case the issue appears to be Postfix's lack of support for the BINARYMIME SMTP extension, causing Postfix to convert all message bodies containing "Content-Transfer-Encoding: 8bit" to BASE64, and breaking DKIM. This only appears to happen when Postfix is an intermediary and the original message was sent to an SMTP server that DOES support BINARYMIME. Doesn't look fixable...

Postfix does not support BINARYMIME at this time because:

• BINARYMIME support would require moderately invasive changes to Postfix, to support email content that is not line-oriented. With BINARYMIME, the Content-Length: message header specifies the length of content that may or may not have line boundaries. Without BINARYMIME support, email RFCs require that binary content is base64-encoded, and formatted as lines of text.
• For delivery to non-BINARYMIME systems including UNIX mbox, the available options are to convert binary content into 8bit text, one of the 7bit forms (base64 or quoted-printable), or to return email as undeliverable. Any conversion would obviously break digital signatures, so conversion would have to happen before signing.

Update: Now I'm not sure about this because the original sending SMTP server doesn't appear to advertise BINARYMIME support.

 

[Bobbbb]

Thanks. But in my case the issue appears to be Postfix's lack of support for the BINARYMIME SMTP extension, causing Postfix to convert all message bodies containing "Content-Transfer-Encoding: 8bit" to BASE64, and breaking DKIM. This only appears to happen when Postfix is an intermediary and the original message was sent to an SMTP server that DOES support BINARYMIME. Doesn't look fixable...



Update: Now I'm not sure about this because the original sending SMTP server doesn't appear to advertise BINARYMIME support.

Yeah, nevermind, that was a wild goose chase. Not dealing with binary here.

 

[Alex Presland]

Since we couldn't change the incoming content, we've opted for adding this to main.cf:

Which fixed the problem, you can choose to have a bigger number instead of 0 (0 disables the limit), maybe that will help you too.

Tried that but sadly DMARC is still showing as broken. :-(
Thanks for the idea...