Question Find source of problem sending e-mail with different HELO value

[Walkum]

Server operating system version

Debian 11

Plesk version and microupdate number

18.0.54

Hello everyone.

Since a week ago we got into the XBL and CSS lists of Spamhaus.org, due to a problem with a mysterious e-mail sending whose HELO value is "outlook.com". This is obviously impossible because on this server we have HELO well configured with the server name, more than 400 email accounts that have been running for 3 years and we have never had this problem until now. It's certainly very strange, it definitely has to be some kind of malware or something similar, and it's related to something in PHP.

I have used ImunifyAV and have all the websites clean.

I'm desperate because I can't find the source of the problem.

How can I find the cause? I have searched the entire forum and the internet and I do not see any case similar to mine.

I am attaching an image of the Spamhaus report and my conversation with one of the agents who attended me.

Thank you so much.

 

[Peter Debik]

Can you identify one or two single mails in /var/log/maillog that are sent out as spam? Are there may pending mails in the mail queue that could be examined further?

 

[Walkum]

Hi again, thanks for the rapid answer.

There are two cases:

First case: E-mail is sended but after a second I receive a MAIL DELIVERY SYSTEM ERROR mail telling me that the mail was not delivered caused by the Spamhaus blacklist. The log of one e-mail rejected sent by Spamhaus:

Jul 22 22:08:32 amberserver postfix/smtpd[3199703]: connect from localhost.localdomain[::1]
Jul 22 22:08:32 amberserver postfix/smtpd[3199703]: TLS SNI localhost from localhost.localdomain[::1] not matched, using default chain
Jul 22 22:08:32 amberserver postfix/smtpd[3199703]: BCE2F33800A8: client=localhost.localdomain[::1], sasl_method=PLAIN, sasl_username=administracion@delope.es
Jul 22 22:08:32 amberserver psa-pc-remote[2154794]: BCE2F33800A8: from=<administracion@delope.es> to=<joseantonio_lg@hotmail.com>
Jul 22 22:08:32 amberserver postfix/cleanup[3203148]: BCE2F33800A8: message-id=<5834ea4b86f131e7c94b68f96e9721ed@delope.es>
Jul 22 22:08:32 amberserver psa-pc-remote[2154794]: BCE2F33800A8: py-limit-out: stderr: INFO:__main__:Setting 'X-PPP-Vhost' header to 'delope.es'
Jul 22 22:08:32 amberserver psa-pc-remote[2154794]: BCE2F33800A8: py-limit-out: stderr: PASS
Jul 22 22:08:33 amberserver psa-pc-remote[2154794]: BCE2F33800A8: spf: stderr: PASS
Jul 22 22:08:33 amberserver psa-pc-remote[2154794]: BCE2F33800A8: check-quota: stderr: SKIP
Jul 22 22:08:33 amberserver psa-pc-remote[2154794]: BCE2F33800A8: dk_sign: stderr: PASS
Jul 22 22:08:33 amberserver postfix/qmgr[2106400]: BCE2F33800A8: from=<administracion@delope.es>, size=1096, nrcpt=1 (queue active)
Jul 22 22:08:33 amberserver postfix/smtpd[3199703]: disconnect from localhost.localdomain[::1] ehlo=1 auth=1 mail=1 rcpt=1 data=1 quit=1 commands=6
Jul 22 22:08:33 amberserver dovecot: imap-login: Login: user=<administracion@delope.es>, method=PLAIN, rip=::1, lip=::1, mpid=3203632, TLS, session=<9pMd8hgBNtkAAAAAAAAAAAAAAAAAAAAB>
Jul 22 22:08:33 amberserver dovecot: service=imap, user=administracion@delope.es, ip=[::1]. Disconnected: Logged out rcvd=977, sent=749
Jul 22 22:08:33 amberserver postfix/smtpd[3203365]: connect from localhost.localdomain[127.0.0.1]
Jul 22 22:08:33 amberserver postfix/smtpd[3203365]: 4C2B133800AB: client=localhost.localdomain[127.0.0.1], orig_queue_id=BCE2F33800A8, orig_client=localhost.localdomain[::1]
Jul 22 22:08:33 amberserver psa-pc-remote[2154794]: 4C2B133800AB: from=<administracion@delope.es> to=<joseantonio_lg@hotmail.com>
Jul 22 22:08:33 amberserver postfix/cleanup[3203148]: 4C2B133800AB: message-id=<5834ea4b86f131e7c94b68f96e9721ed@delope.es>
Jul 22 22:08:33 amberserver psa-pc-remote[2154794]: 4C2B133800AB: py-limit-out: stderr: INFO:__main__:No SMTP AUTH and not running in sendmail context (incoming or unrestricted outgoing mail). SKIP message.
Jul 22 22:08:33 amberserver psa-pc-remote[2154794]: 4C2B133800AB: py-limit-out: stderr: SKIP
Jul 22 22:08:33 amberserver psa-pc-remote[2154794]: 4C2B133800AB: spf: stderr: PASS
Jul 22 22:08:33 amberserver psa-pc-remote[2154794]: 4C2B133800AB: check-quota: stderr: SKIP
Jul 22 22:08:33 amberserver psa-pc-remote[2154794]: 4C2B133800AB: dk_sign: stderr: PASS
Jul 22 22:08:33 amberserver postfix/qmgr[2106400]: 4C2B133800AB: from=<administracion@delope.es>, size=2634, nrcpt=1 (queue active)
Jul 22 22:08:33 amberserver postfix/smtpd[3203365]: disconnect from localhost.localdomain[127.0.0.1] ehlo=1 xforward=1 mail=1 rcpt=1 data=1 quit=1 commands=6
Jul 22 22:08:33 amberserver amavis[2935938]: (2935938-17) Passed CLEAN {RelayedInbound}, [::1]:42710 <administracion@delope.es> -> <joseantonio_lg@hotmail.com>, Queue-ID: BCE2F33800A8, Message-ID: <5834ea4b86f131e7c94b68f96e9721ed@delope.es>, mail_id: aTJEqrD22qWV, Hits: -1.108, size: 1745, queued_as: 4C2B133800AB, 387 ms
Jul 22 22:08:33 amberserver postfix/smtp[3203158]: BCE2F33800A8: to=<joseantonio_lg@hotmail.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=0.73, delays=0.33/0/0/0.39, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 4C2B133800AB)
Jul 22 22:08:33 amberserver postfix/qmgr[2106400]: BCE2F33800A8: removed
Jul 22 22:08:33 amberserver dovecot: imap-login: Login: user=<administracion@delope.es>, method=PLAIN, rip=::1, lip=::1, mpid=3203643, TLS, session=<GLIo8hgBQtkAAAAAAAAAAAAAAAAAAAAB>
Jul 22 22:08:33 amberserver dovecot: service=imap, user=administracion@delope.es, ip=[::1]. Disconnected: Logged out rcvd=74, sent=2158
Jul 22 22:08:34 amberserver postfix/smtpd[3196269]: lost connection after AUTH from unknown[46.148.40.199]
Jul 22 22:08:34 amberserver postfix/smtpd[3196269]: disconnect from unknown[46.148.40.199] ehlo=1 auth=0/1 rset=1 commands=2/3
Jul 22 22:08:34 amberserver dovecot: imap-login: Login: user=<administracion@delope.es>, method=PLAIN, rip=::1, lip=::1, mpid=3203645, TLS, session=<dV4t8hgBRNkAAAAAAAAAAAAAAAAAAAAB>
Jul 22 22:08:34 amberserver dovecot: service=imap, user=administracion@delope.es, ip=[::1]. Disconnected: Logged out rcvd=502, sent=35019
Jul 22 22:08:34 amberserver postfix/smtp[3203169]: 4C2B133800AB: to=<joseantonio_lg@hotmail.com>, relay=hotmail-com.olc.protection.outlook.com[104.47.56.161]:25, delay=1.4, delays=0.17/0/1.1/0.15, dsn=5.7.1, status=bounced (host hotmail-com.olc.protection.outlook.com[104.47.56.161] said: 550 5.7.1 Service unavailable, Client host [51.15.185.237] blocked using Spamhaus. To request removal from this list see https://www.spamhaus.org/query/ip/51.15.185.237 (AS3130). [CO1NAM11FT023.eop-nam11.prod.protection.outlook.com 2023-07-22T20:08:34.625Z 08DB8AD501D09E7F] (in reply to MAIL FROM command))
Jul 22 22:08:34 amberserver postfix/smtp[3203169]: 4C2B133800AB: lost connection with hotmail-com.olc.protection.outlook.com[104.47.56.161] while sending RCPT TO
Jul 22 22:08:34 amberserver postfix/cleanup[3203148]: B4CFA33800AE: message-id=<20230722200834.B4CFA33800AE@amberserver.es>
Jul 22 22:08:34 amberserver postfix/bounce[3203170]: 4C2B133800AB: sender non-delivery notification: B4CFA33800AE
Jul 22 22:08:34 amberserver postfix/qmgr[2106400]: B4CFA33800AE: from=<>, size=5319, nrcpt=1 (queue active)
Jul 22 22:08:34 amberserver postfix/qmgr[2106400]: 4C2B133800AB: removed
Jul 22 22:08:34 amberserver postfix-local[3203647]: B4CFA33800AE: from=<MAILER-DAEMON>, to=<administracion@delope.es>, dirname=/var/qmail/mailnames
Jul 22 22:08:34 amberserver dk_check[3203649]: B4CFA33800AE: DKIM Feed: No signature
Jul 22 22:08:34 amberserver postfix-local[3203647]: B4CFA33800AE: dk_check: stderr: PASS
Jul 22 22:08:34 amberserver dmarc[3203650]: B4CFA33800AE: SPF record was not found in Authentication-Results
Jul 22 22:08:34 amberserver postfix-local[3203647]: B4CFA33800AE: dmarc: stderr: PASS
Jul 22 22:08:34 amberserver dovecot: service=lda, user=administracion@delope.es, ip=[]. Warning: Failed to parse return-path header
Jul 22 22:08:34 amberserver dovecot: service=lda, user=administracion@delope.es, ip=[]. sieve: msgid=<20230722200834.B4CFA33800AE@amberserver.es>: stored mail into mailbox 'INBOX'
Jul 22 22:08:34 amberserver postfix/pipe[3203171]: B4CFA33800AE: to=<administracion@delope.es>, relay=plesk_virtual, delay=0.15, delays=0.01/0/0/0.13, dsn=2.0.0, status=sent (delivered via plesk_virtual service)
Jul 22 22:08:34 amberserver postfix/qmgr[2106400]: B4CFA33800AE: removed
Jul 22 22:08:35 amberserver dovecot: imap-login: Login: user=<administracion@delope.es>, method=PLAIN, rip=::1, lip=::1, mpid=3203657, TLS, session=<DoU88hgBqJkAAAAAAAAAAAAAAAAAAAAB>
Jul 22 22:08:35 amberserver dovecot: service=imap, user=administracion@delope.es, ip=[::1]. Disconnected: Logged out rcvd=130, sent=2216


Second case: The e-mail is not sended and is in the mail pending queue. I attach a image when use the command "postqueue -p".

These examples are of normal e-mails sent through webmail. I do not identify in the logs any email sent with HELO value "outlook.com" as indicated in the screenshot of my previous Spamhaus post.


Thanks.

 

[Peter Debik]

Very likely thee mails are submitted to the MTA by your login of your email address administrator@.... To mitigate the issue, it will probably be sufficient to change the password of your mailbox. The login credentials must be stored in one of your websites. Either in a configuration file or a database (both based on some plugin). I have most frequently seen this for Joomla installations, occasionally for Wordpress installations. But it may not be limited to these two packages. Malicious plugins of a CMS can abuse such stored SMTP login data to abuse the SMTP server for sending spam. When you change the mailbox password on your server, the login credentials will no longer work, and that means that the malicious software cannot login to the mail server any longer. Please check this approach first. I'd also recommend to search your websites for stored SMTP login data. Please report here if changing the password does not solve the problem.

 

[Walkum]

Hi again, thanks for the response.

Indeed it is confirmed that there are mail sends from the server with port 25 (supposedly) and with that incorrect HELO value, they have put us on new blacklists (Abusix) confirming this in more detail than the Spamhaus guys.

I have been told to use the command "sudo ss -tnp '( dport = 25 or dport = 587 )'" but it not shows anything.

The max information I can provide is: I have all WordPress updated, email passwords changed, and on approximately 5 or 6 websites I use PHPMailer, the latter was out of date since 2021 and I have updated it to the latest version.

The most important thing that I need to know, is if there is a way to find out the root problem through some debugging method or something similar, in /var/log/maillog I don't see anything interesting.

Thank you very much and sorry for the headache.

 

[Walkum]

Both the guys from Abusix and Spamhaus tell me to block outgoing port 25 on the server, but I've read on the forum that this prevents me from sending e-mails to hosts like Gmail or similar. How can I make this recommendation without negatively affecting it? Thank you

 

[Peter Debik]

Jul 22 22:08:32 amberserver postfix/smtpd[3199703]: connect from localhost.localdomain[::1]
...
Jul 22 22:08:32 amberserver postfix/smtpd[3199703]: BCE2F33800A8: client=localhost.localdomain[::1], sasl_method=PLAIN, sasl_username=administracion@delope.es

indicates that a website logins using mailbox login and password of administracion@delope.es. Else, if a website used the PHP mail() function without SMTP login, it would read

Jul 22 22:08:32 amberserver postfix/pickup[3199703]: connect from localhost.localdomain[::1]

Are you sure that you changed all passwords of all mailboxes and that you are not using SMTP login algorithms, e.g. through PHPMailer? Updating WP, plugins etc. is a good idea, but if you have a malicious script somewhere, it won't help. If you have update the SMTP login data in your websites, too, then the malicious code an again use that login and continue sending spam.

 

[Walkum]

I changed the fail2ban configuration making it more strength (especially for postfix), I have manually banned the suspicious IP's that I have seen in the logs, and I have deleted the plugins that had vulnerabilities related to emails that did not have an update in the official WordPress repository with fixes for these vulnerabilities.

It seems miraculous but we have been removed from the lists this morning. It seems that it has worked.

ImunifyAV hasn't detected any malware for me on any website so if there is, I can't think of what else I could use to find it.

Thanks so much for the help. If I see that I enter the lists again, I will get in touch again.

Cheers and thank you again.

 

[Maarten]

Malware often nestles itself in the /tmp or /var/tmp directories. You can check there for strangely named files or hidden files:

# ls -la /tmp
# ls -la /var/tmp

You can also check for (hidden) files that are changed since the issue began:

# cd /var/www/vhosts/example.com

Check for files changed/created in the last 7 days:

# find . -mtime -7 -ls

Check for hidden directories changed/created in the last 7 days:

# find . -mtime -7 -path "*/.*" -ls

 

[Walkum]

This is the unique apparently strange file, I dare not delete it in case it is something from the system. The last modification is from May 18. You sound familiar? Are you part of Plesk? It's es83G3KK file. Attached image.

 

[Maarten]

That file is part of the plesk-installer (it has the same date as the plesk-installer.lock).
Either way, it doesn't have execution rights, so this is harmless.

 

[Walkum]

Ok, thank you. Apparently there are not any malicious file on the system, so I think that I can be quiet for now.

Thanks Maarten. and Peter for your help

 

[Walkum]

Again listed on Spamhaus, 24 hours later, appears to be a joke. This is a darkness. I don't know how this can happens. There aren't any malware on any site. PHPMailer updated and secured with all the passwords. This is very weird.

 

[Walkum]

The reason for the lists again is because of an e-mail sent with HELO outlook.com:

51.15.185.237 2023-07-24 17:10:00 outlook.com

I found some lines like this on maillog:

Jul 24 12:38:11 amberserver postfix/smtp[3674547]: B2757337B1A9: to=<contabilidad@avalonbc.es>, relay=avalonbc-es.mail.protection.outlook.com[52.101.68.0]:25, delay=0.87, delays=0.23/0/0.17/0.47, dsn=2.6.0, status=sent (250 2.6.0 <!&!AAAAAAAAAAAuAAAAAAAAAA1rCTS/Bf1KlW0TnddMeyoBAMO2jhD3dRHOtM0AqgC7tuYAAAAAAA4AABAAAACbUYnuMvkKRoX46NHb4HHyAQAAAAA=@junkburger.com> [InternalId=109422881826554, Hostname=DBAPR04MB7205.eurprd04.prod.outlook.com] 286846 bytes in 0.089, 3142.690 KB/sec Queued mail for delivery)
Jul 24 12:38:11 amberserver postfix/qmgr[2106400]: B2757337B1A9: removed

Maybe is util information. Any idea?

 

[Walkum]

RESOLVED.

I will explain how I have solved it to help future people who may have this problem.

First, using the "htop" command (install with "sudo apt-get install htop" if you don't have it), I filtered processes by "cron" name, there were numerous processes like ./cron.php -p0.0.0.0 -e18924 or similar, you have to kill them all. They come from malware.

Second, using the command "netstat -natup | grep :25", all the local connections from the server's IP with port 25 to different external IP's, I killed all those processes. If you look for those external IP's to which a connection is made in MXtoolbox in "Blacklists", you will see that they are on more than 6 blacklists. They are bad IP's, probably from bots.

Finally. I configured fail2ban so that specifically for "plest-postfix" it bans when detecting 3 errors, with a duration of 3 hours (this parameter can be configured to be like as you want). Without exaggeration, 1500 IP's fell in a few hours, all from bots. Don't be afraid of false positives.

With all this steps, no longer more Spamhaus included me on blacklists.

Obviously, try check all malware that would be in your server and clean it before doing all this steps.

Hope that helps.

All the best.