Forwarded to devs DMARC Changes in MU#51 Introduce New Segfault Error

[G J Piper]

TITLE:

DMARC Changes in MU#51 Introduce New Segfault Error​

PRODUCT, VERSION, OPERATING SYSTEM, ARCHITECTURE:

Google Compute Engine server running CentOS 7.6.1810 and Plesk 7.8.11#51 with postfix 2.10.1 and dovecot 2.3.4.1​

PROBLEM DESCRIPTION:

I'm not seeing any email failures to deliver (yet) however in my /var/log/messages log I'm getting a lot of DMARC repeating errors:

Apr 22 23:03:58 pcs-plesk-centos7-web-vm kernel: dmarc[4945]: segfault at 0 ip 00007feedead4de0 sp 00007ffc866f5e08 error 4 in libc-2.17.so[7feede996000+1c2000]
Apr 22 23:04:02 pcs-plesk-centos7-web-vm kernel: dmarc[4981]: segfault at 0 ip 00007f446d5ddde0 sp 00007ffc836d0488 error 4 in libc-2.17.so[7f446d49f000+1c2000]

View attachment 15470​

STEPS TO REPRODUCE:

Have a postfix and dovecot server configured in Plesk with SPF, DKIM, and DMARC protections all enabled.

​

ACTUAL RESULT:

Errors appeared in logs after latest update to MU#51 whose release notes include:

• Emails received from senders with long names (when the “From” field takes several lines) no longer cause a DMARC error ending up in /opt/psa/handlers/spool by mistake. (PPPM-7163)
• Emails from amazon.de and husqvarnagroup.com now pass the DMARC verification and do not go to spam. (PPPM-6847)

EXPECTED RESULT:

No errors please. ​

ANY ADDITIONAL INFORMATION:

​

YOUR EXPECTATIONS FROM PLESK SERVICE TEAM:

Confirm bug​

 

[IgorG]

Could not reproduce the issue on a test environment with the same Plesk version and OS.

Additional information is required:

1. Does the issue occur for all incoming mail?
2. Provide with the header of the email that triggers the error

 

[G J Piper]

Could not reproduce the issue on a test environment with the same Plesk version and OS.

Additional information is required:

1. Does the issue occur for all incoming mail?

No.

Could not reproduce the issue on a test environment with the same Plesk version and OS.

1. Provide with the header of the email that triggers the error

I couldn't get the headers from a particular email for this, but I don't think they will help as much as the logs anyway. I'm attaching a screenshot of the logs as I've been able to collect from the exact same time they happened. There seems tyo be an error in the maillog that corresponds to the dmarc error in the messages log, as follows:

Notice:

Apr 24 20:36:37 pcs-plesk-centos7-web-vm postfix-local[28715]: _mh_fork(): The child process with pid 28719 killed by signal 11

at the same time as the

Apr 24 20:36:37 pcs-plesk-centos7-web-vm kernel: dmarc[28719]: segfault at 0 ip 00007f7c4ebd3de0 sp 00007ffd8aceedf8 error 4 in libc-2.17.so[7f7c4ea95000+1c2000]

Messages log of the dmarc error:

Mail Log of the same second:


This screenshot will give you an idea of the frequency:

 

[G J Piper]

A ha! I found the email (I think) associated with the logs I showed you in the last post:
(It was sent from an iPhone as an outgoing email, in which this is the BCC back to itself BTW)

Maybe it only happens with emails that BCC the from address?

 

[G J Piper]

Apparently this is an old problem that was once "fixed" but it is baaaaaack!
DMARC is segfaulted permanently: _mh_fork(): The child process with pid XXXX killed by signal 11

This page shows this issue as fixed: Plesk Onyx 17.8.11 Update 51 22 April 2019 (Linux)

However, I didn't begin having this problem until MU#51, which they say is a "fixed" version.

Upon checking the other symptoms listed on that page, I find this directory on my server is indeed full of little files as well: /usr/local/psa/handlers/spool/

Their fix caused the issue it claims to fix for some people?

I've since had some complaints from customers saying some of their emails are not getting to them. I've unchecked the "Enable DMARC to check incoming mail" setting until this gets fixed or there is a workaround published. Maybe they can tell us what the "fix" was they applied so we can try to undo it or retry it?

 

[G J Piper]

Two more notes to add:

1. When "Enable DMARC to check incoming mail" is turned off, the /usr/local/psa/handlers/spool/ directory remains empty
2. With DMARC enabled, emails fill /usr/local/psa/handlers/spool/ but they are still being delivered.

 

[Adam Dobruczky]

I can confirm this promlem too on several servers!!!

PRODUCT, VERSION, OPERATING SYSTEM, ARCHITECTURE:
Dedicated server running Debian 9 and Plesk Onyx v17.8.11_build1708180301.19 os_Debian 9.0 with postfix 3.1.12-0+deb9u1 and dovecot 2.3.4.1-debian9.0.19031416

 

[IgorG]

From developers:

Cannot reproduce, provided information is insufficient (including the one in the forum thread).
Segmentation faults are notoriously hard to investigate w/o direct access, so my only proposition for the customer is to open a support ticket and provide access to their server.

 

[G J Piper]

From developers:

Cannot reproduce, provided information is insufficient (including the one in the forum thread).
Segmentation faults are notoriously hard to investigate w/o direct access, so my only proposition for the customer is to open a support ticket and provide access to their server.

Ugh... I cannot let someone mess around in my active production server while hosted customers are actively using it... The devs can surely set up a free 30-day trial of my exact server online though and do their own — it is just a Google Compute Engine Plesk Onyx Webhost Edition found as a standard supported setup on this page of Plesk Documentation. If they need SMTP for it like mine it is using SendGrid's SMTP service tied into postfix.

 

[G J Piper]

I actually set up a second server as described in my previous post. I then restored an old snapshot from a while back into it, removed all the domains, installed a couple of unused ones I had lying around, and get it all set up like my current server. I then looked through the logs and indeed did not find any of those DMARC errors! Then I looked at the Plesk version: Version 17.8.11 Update #46

So, I went in and upgraded Plesk to the current MU#53 using this command:

plesk installer --select-release-current --reinstall-patch --upgrade-installed-components

Immediately after, I looked through the logs and voila — several DMARC errors are now coming in just like my current production server. It appears easily replicated, and I can give someone access to this server if they wish. However, I believe they would have better results starting their own since it is supposedly a supported "push-button" server setup in Google Cloud, and see what they find. It is free to set up a trial server at Google Compute Engine: Google Compute Engine Plesk Onyx Webhost Edition

Can't they just open one and examine it? Maybe @IgorG you could open one and see if you get the same results in your messages log?

 

[G J Piper]

Another small tidbit I discovered on my backup server... Maybe @Adam Dobruczky could verify on one of his servers:
The dmarc error only seems to occur on incoming emails that are forwarded internally to another email address on the same server (maybe different domain only -- haven't verified that).

Also, in my old mail log, I never got these messages until the new update to MU#53 set loose the dmarc errors too:

dmarc[17485]: Store DKIM result for '<anydomain.ext>' into DMARC library.

Each occurrence of that mail log entry is accompanied at exactly the same time by the error message in the messages log.

Having an older server to compare to, with little to no email activity except my tests is a revealing thing!

 

[Adam Dobruczky]

Hi,

Yes, i can confirm too, that dmarc error only occurs on incoming emails that are forwarded internally.

messages log:

May 30 11:17:34 xxx kernel: [1463593.820393] dmarc[24883]: segfault at 0 ip 00007f82e63240b0 sp 00007ffe8584b0e8 error 4 in libc-2.24.so[7f82e6203000+195000]

maillog:

Line 19053: May 30 11:17:34 xxx dmarc[24883]: Starting the dmarc filter...
Line 19054: May 30 11:17:34 xxx postfix-local[24874]: _mh_fork(): The child process with pid 24883 killed by signal 11
Line 19055: May 30 11:17:34 xxx postfix-local[24874]: Error during 'dmarc' handler

 

[Adam Dobruczky]

Hi, G J Piper, and support team!

I found this thread,:
DMARC check fails: Unable to store SPF/DKIM results into DMARC library

and because i was desperate, i tried to use the ub16_p175_dmarc.tar.gz on my test server (debian 9.9, not ubuntu), and voila tha kernel segfault error has gone. I will test it because it was made for ubuntu, but im happy to get a rid of this error.

Grep please try it, ans test it.

 

[G J Piper]

Ok I have done what I asked the Plesk Devs to do, and have reproduced (reliably) the problem. Devs, Support Team, please do these instructions to verify the problem:

FULL INSTRUCTIONS FOR REPRODUCING THE ERRORS

Create Google Cloud Plesk Onyx CentOS Deployment found here:
Google Cloud Platform

In Google Instance, reserve static IP address.

Log in to new default install of Plesk Panel, then do the following:

1) Change Password

2) "Update All" packages

3) Setting Adjustment: Tools & Settings -> DNS Template -> "Switch Off", "Apply changes to all hosted domains - All Zones"

4) Set up a new domain (domain.com) on "Default Simple" subscription.

5) Setting Adjustment: Modify DNS settings in Subscriptions. Enable DNS, then switch to "slave" setting.

6) Set up Remote DNS (I use GoDaddy DNS) with these settings: (not sure if this is needed to reproduce the error, but it is how I did it)
----------------------------------------------------
domain.com
A @ 123.123.123.123 1 Hour
A webmail 123.123.123.123 1 Hour
A www 123.123.123.123 1 Hour
MX 10 @ domain.com 1 Hour
----------------------------------------------------

7) Set up two email accounts on the domain.com domain: [email protected] and [email protected] with default settings.

8) Set [email protected] to forward to [email protected]

9) Setting Adjustment: Tools & Settings -> Mail Server Settings -> Default settings, but turn on the following:
"Enable SPF spam protection to check incoming mail ON"
"DKIM spam protection: Allow signing outgoing mail ON and Verify incoming mail ON"
"Enable DMARC to check incoming mail ON"
"Switch on limitations on outgoing email messages ON" (CRITICAL DISCOVERY: WITH THIS TURNED OFF, THERE ARE NO ERRORS!)

10) SSH into the server and observe the "messages" and "maillog" log using this command: "tail -n777 -f /var/log/maillog -f /var/log/messages"

11) Send an email to [email protected] and observe the errors.

12) If you like, switch off "Switch on limitations on outgoing email messages" and observe that the errors go away. Turning it back on resumes the errors.


Please fix this for us.

 

[Adam Dobruczky]

i want to complement what G J Piper has been said:

• The problem is OS independent: we found this error on CentOS, and Debian 9.9 too.
• You dont need to forward an email from one domain to another, forwarding to the same domain generates error too. Important is that the forwarding has to be on the same server.
• The forwarded emails arrives, but not deleted from /opt/psa/handler/spool/
• I think that this Segfault error and this thread (DMARC check fails: Unable to store SPF/DKIM results into DMARC library) have to do with each other.

I have changed the dmarc file on a test server (debian 9), with this ub16_p175_dmarc.tar.gz, The limitations on outgoing email messages, was not chaned so it remains on, and the error has gone, please consider it, when you investigate this.

 

[G J Piper]

You don't need to forward an email from one domain to another, forwarding to the same domain generates error too.

You are correct. You can forward email all within one domain and it still gives the error. I'll simplify my instructions above. If possible, can someone else please try the instructions I made and see if they get my results? Devs say they did, but couldn't see any errors.

 

[Adam Dobruczky]

Yes, we got a new ticket:
DMARC segfault errors in server logs: _mh_fork(): The child process with pid XXXXX killed by signal 11

 

[G J Piper]

It seems like a logical solution for this would be to not launch the DMARC handler at all if the email is coming from, or forwarded from, the same server...

 

[TomBoB]

Can also confirm above case.

var/log/messages

Jun 30 20:22:51 rhino kernel: dmarc[16482]: segfault at 0 ip 00007f31a1bd25d0 sp 00007fff082e1028 error 4 in libc-2.17.so[7f31a1b46000+1c2000]

maillog

Jun 30 20:22:51 rhino spamd[594]: spamd: clean message (-1.9/4.5) for [email protected]:30 in 0.8 seconds, 28605 bytes.
Jun 30 20:22:51 rhino spamd[594]: spamd: result: . -1 - BAYES_00,DKIMWL_WL_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,T_KAM_HTML_FONT_INVALID,URIBL_BLOCKED scantime=0.8,size=28605,[email protected],uid=30,required_score=4.5,rhost=rhino.ourserver.com,raddr=::1,rport=54974,mid=<[email protected]>,bayes=0.000000,autolearn=ham autolearn_force=no
Jun 30 20:22:51 rhino autoresponder[16480]: Starting the autoresponder filter...
Jun 30 20:22:51 rhino autoresponder[16480]: Exiting. No need to respond.
Jun 30 20:22:51 rhino autoresponder[16480]: Unable to write message into stdout
Jun 30 20:22:51 rhino postfix-local[16465]: _mh_fork(): The child process with pid 16480 killed by signal 6
Jun 30 20:22:51 rhino postfix-local[16465]: Error during 'autoresponder' handler
Jun 30 20:22:51 rhino dk_check[16481]: Starting the dk_check filter...
Jun 30 20:22:51 rhino dk_check[16481]: DKIM verify result: Success
Jun 30 20:22:51 rhino dmarc[16482]: Starting the dmarc filter...
Jun 30 20:22:51 rhino postfix-local[16465]: _mh_fork(): The child process with pid 16482 killed by signal 11
Jun 30 20:22:51 rhino postfix-local[16465]: Error during 'dmarc' handler
Jun 30 20:22:51 rhino spamd[30671]: prefork: child states: II
Jun 30 20:22:51 rhino dovecot: service=lda, [email protected], ip=[]. msgid=<[email protected]>: saved mail to INBOX

CentOS Linux 7.6.1810 (Core)‬; Plesk Onyx Version 17.8.11 Update #58

Production servers with very similar setup as G J Pipers test setup above. We're also using external DNS.

Hope it helps pin it down and resolve.

 

[IgorG]

Guys, thank you for the additional details which allowed us to reproduce the issue, confirm it and submit as PPPM-10547