Resolved Preventing bot brute force probes against Postfix/SMTP

iainh

Regular Pleskian
Server operating system version
AlmaLinux 9.8 (Olive Jaguar)
Plesk version and microupdate number
Plesk Obsidian 18.0.79 Update #3

Before we start...

I am posting this more as advice to others. It could be considered in tandem with my post 'Plesk firewall and preventing clashes with firewalld' and generally hardening a Plesk server.

This firewallld post is important IMHO as I found admin services with access locked-down via the Plesk firewall being endlessly probed. This was becasue the Plesk firewall config was being bypassed due to the server having been built by the hosting company with firewalld active and adding it's own configuration. There’s a lesson here: Just because you have configured access in the Plesk firewall does not mean this is the access being enforced if there is some other configuration, like from a default firewallld installation by your hosting company, also operating. My advice: Go check whether what you think is operating, is actually operating. I found a horror story of attempts against SSH which should have been impossible according to the Plesk firewall config … all due to 'interference' from this default firewallld configuration that I was blissfully unaware of.

Postfix and SMTP brute force attacks

Moving on now to Postfix and SMTP brute force attacks, I was finding my maillog filled with huge numbers of log lines similar to:
Code:
[date timestamp] mail postfix/smtpd[task_id]: warning: unknown[ip_address]: SASL LOGIN authentication failed: authentication failure, sasl_username=[some_username]

If you’d quite like to stop this endless probing of your mail accounts, then this post is for you.

Finding the scale of probing against your server

If you don’t know whether your maillog is filled with these login probes but are now curious, then login (SSH or console) and:
Code:
cd /var/log
grep "authentication failed" maillog | more

You will likely get screenfuls of lines of random IPs from all over the world, all attempting logins using a mix of real and non-existent usernames and Email addresses. Want to see even more? Then try:
Code:
grep "authentication failed" maillog* | more

and you’ll get the output from both the maillog and maillog.processed files. And you can get even more by looking back through the compressed maillog archives with:
Code:
gunzip -c maillog*.gz | grep "authentication failed" | more


Stopping the probes​

In the fair assumption you’ll find the logs filled with these endless probes, the next question is: So, can I stop this noise and what is effectively people endlessly ‘rattling the locks’? The answer is yes…

Background​

Originally all mail sent was via port 25 (SMTP), but these days clients are (hopefully) configured to use either port 465 or 587 so that mail transmission is encrypted. I find MS Outlook will accept you configuring 587 without error or warning but silently changes the connection without telling you to 465! Anyway, 465 is reserved for secure email transmission using implicit SSL/TLS encryption known as SMTPS, while port 587 begins unencrypted and upgrades to TLS via STARTTLS. In contrast, port 465 encrypts from the very first packet. Whichever you configure, you should submit mail to your mail server using one of these ports and not 25.

The significance of this is that clients shouldn’t be attempting authentication to port 25 which is now generally considered as for MTA to MTA (server to server) transmission only. This in turn means attempts to authenticate against port 25 will not be legitimate as your legitimate clients should be authenticating via 465 or 587.

A game of two halves​

There are two parts to what follows:
  1. Disabling authentication against port 25
  2. Using fail2ban to block failed login attempts against Postfix operating your smptd on port 25
Part 1 isn’t essential, but it means no probe can ever be successful (on port 25, the port favoured by the bots as it's unencrypted) even with the correct credentials as authentication isn’t permitted and simply returns a 503 5.5.1 Error: authentication not enabled error to the bot, as in:

Code:
>> telnet [your_server] 25
<< 220 [your_server] ESMTP Postfix
>> EHLO [your_device_name]
<< 250-[your_server]
<< 250-PIPELINING
<< 250-SIZE 30720000
<< 250-ETRN
<< 250-STARTTLS
<< 250-ENHANCEDSTATUSCODES
<< 250-8BITMIME
<< 250-DSN
<< 250 CHUNKING
>> AUTH LOGIN
<< 503 5.5.1 Error: authentication not enabled

Disabling authentication to port 25​

You need to edit the configuration file /etc/postfix/master.cf. Change:
Code:
smtp       inet  n       -       n       -       -       smtpd

To read:
Code:
smtp       inet  n       -       n       -       -       smtpd
  -o smtpd_sasl_auth_enable=no
(sorry, I would include the comment lines above this line to make recognition simpler, but including these generates an error in the text editor here as does including the block even as an image. An 'undocumented feature' of the editor!)
The '-o’ option should be on a separate line and must be indented by at least one space to be recognised as an option of the smtp declaration on the line above.

After making this change, restart Postfix:
Code:
sudo systemctl restart postfix

You can now go and test whether authentication on port 25 is permitted or not by trying the telnet [your_server] 25 check shown above. Presuming you get the ‘503 5.5.1 Error: authentication not enabled’ response back, you know that no amount of probing will ever give up any details about a good or bad credential when tested on port 25, the port of choice for this kind of probing (as it doesn’t require establishing a TLS session). However, this change hasn’t stopped the probes being attempted nor the log lines being recorded, just made any amount of probing ineffective.

Stopping the probes

This is where we use fail2ban to actually stop the probes and associated log lines. This can be done with or without the disabling of authentication against port 25 outlined above.

An 'interesting' discovery was that despite hav ing (sorry, have to include that space, but if typed properly the editor blows-up again!) the ‘plesk-postfix’ jail enabled in Plesk (Home > Tools & Settings > IP Address Banning > Jails), there were never an IPs listed despite the volume of failed auth attempts. The reason is the default Plesk fail2ban filter (that picks out failed auth attempts from maillog and so ‘drives’ fail2ban) does not detect failed login attempt log lines … it’s patterns don’t match.

Rather than mess with the default Plesk fail2ban filter which picks out other things other than just failed auth attempts, I created an additional ‘override file’: /etc/fail2ban/filter.d/postfix.local. This should contain only (meaning clear out any default config you might find in it):

Code:
[Definition]
# 1. Define your custom rule here safely
mdre-auth3 = ^%%(__prefix_line)swarning: unknown\[<HOST>\]: SASL LOGIN authentication failed: authentication failure, sasl_username=.*$

# 2. Reference the exact underlying macro engine token to prevent the loop
failregex = <mdre-<mode>>
            %(mdre-auth3)s

The use of ‘mdre-auth3’ is that the mail filer contains definitions for both ‘mdre-auth’ and ‘mdre-auth2’, so this just extended that pattern. The mdre-auth3 pattern matches the log line format found in maillog. Having saved the override file, now check it with:
Code:
sudo fail2ban-regex systemd-journal /etc/fail2ban/filter.d/postfix.conf

You should see a clean, successful compilation, basically adding in the new auth failure pattern that this time does match that Postfix output/maillog log lines. Now restart fail2ban:
Code:
sudo systemctl restart fail2ban

There are now two tests that your handiwork is working:
Code:
iptables -L f2b-plesk-postfix -n -v

gives you a list of banned IPs. The list will grow over time, but looks something like:
Code:
Chain f2b-plesk-postfix (1 references)
pkts bytes target prot opt in out source destination
0 0 REJECT all -- * * 78.66.44.246 0.0.0.0/0 reject-with icmp-port-unreachable
0 0 REJECT all -- * * 205.209.65.70 0.0.0.0/0 reject-with icmp-port-unreachable
0 0 REJECT all -- * * 87.236.176.143 0.0.0.0/0 reject-with icmp-port-unreachable
…
15921 15M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0

Prior to the fail2ban changes outlines above, you’ll only see the final ‘RETURN’ line, meaning nothing is matching, everything is being passed by fail2ban.

Your other check is:
Code:
fail2ban-client status plesk-postfix

This will give you an output like:
Code:
Status for the jail: plesk-postfix
|- Filter
|  |- Currently failed: 136
|  |- Total failed:     684
|  `- File list:        /var/log/ maillog
`- Actions
   |- Currently banned: 268
   |- Total banned:     279
   `- Banned IP list:   111.70.38.54 71.235.225.181 163.61.33.117 ...
(note: there should be no space in the file list path, but formatting it properly just causes the editor here to blow up with "Oops! We ran into some problems. Please try again later. More error details may be in the browser console.")

The IP list will grow and grow. This is fail2ban doing it’s stuff.

The final part is to tighten up fail2ban, as remember, there should be no port 25 auth attempts, but that will need to be posted as a reply to this as the editor will not allow me to add more!
 
Right, to the final part ... and maybe the editor will play more nicely now??

We now have authentication attempts against port 25 blocked, fail2ban now recognising auth failures being recorded in maillog and blocking bad actors. The plesk-postfix jail is now actually working :)

Our last task is to tighten up fail2ban.

In the Plesk panel/UI, navigate to Home > Tools & Settings > IP Address Banning > Jails > plesk-postfix and click on ‘Change settings’. You will want to review/change:

plesk-postfix jail settings.png

  1. The filter: Click the drop-down and select ‘postfix[mode=aggressive]’. This means fail2ban will be triggered by all the potential failure patterns, e.g. auth failures, DDoS attempts, rBL failures etc.
  2. IP address ban period: Increase the ‘IP address ban period’ from the default of 600 (10 mins) to 'something longer'. You can insert whatever you like, but 84600 is a one day ban after which an IP will be unbanned
  3. The maximum number of failed login attempts: This is defaulted to 5, but you can drop to 3 or even 2 … remembering you are picking up bots and your configured users shouldn’t be recording lots of auth errors and you can always trust their IP if they get themselves in a pickle
Now sit and watched the banned IP list build up.

I hope this helps someone and happy hardening :)



And as the editor seems to be playing more nicely now, let me try and include a larger clip of /etc/postfix/master.cf before and after update mentioned above:

Before:
Code:
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (no)    (never) (100)
# ==========================================================================
smtp       inet  n       -       n       -       -       smtpd

After:
Code:
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (no)    (never) (100)
# ==========================================================================
# [change-date] Added "-o smtpd_sasl_auth_enable=no" option to block authenitcation to Postfix on port 25
#               Revert by removing "-o smtpd_sasl_auth_enable=no" and restarting Postfix (systemctl restart postfix)
smtp       inet  n       -       n       -       -       smtpd
  -o smtpd_sasl_auth_enable=no

Now, why wouldn’t it do that before! :confused:
 
In my initial post:
The use of ‘mdre-auth3’ is that the mail filer contains definitions...

Should read:
The use of ‘mdre-auth3’ is that the jail filter contains definitions...
 
An interesting observation on the auth attempts still being made:
Code:
Jul 22 21:01:23 mail postfix/smtpd[1345648]: warning: dsl-hkibng42-50dfc9-173.dhcp.inet.fi[80.223.201.173]: SASL LOGIN authentication failed: authentication failure, [email protected]
Jul 22 21:01:34 mail postfix/smtpd[1345648]: warning: www.metmalawi.gov.mw[168.253.224.242]: SASL LOGIN authentication failed: authentication failure, sasl_username=info

This pattern of [[email protected]] followed by just [address] from a completely different host is repeated over and over. Each time, [[email protected]] followed by just [address] a few seconds later from an entirely different host, clearly working together. And the two hosts and credentials tried are different each time, so not two hosts working together, but many pairs, all trying different [[email protected]] followed by [address] combinations. Maybe all one network of compromised hosts all under one C2 (command & control)
 
And another observation for those interested in seeing this all working: fail2ban-client status plesk-postfix is showing me a list ending 168.253.224.242 211.216.58.204.

But tailing maillog for "authentication failed:" shows the last entries as:

Code:
Jul 22 21:01:34 mail postfix/smtpd[1345648]: warning: www.metmalawi.gov.mw[168.253.224.242]: SASL LOGIN authentication failed: authentication failure, sasl_username=info

Note the auth failures log ends with 168.253.224.242, while the block list also adds 211.216.58.204.

grep "211.216.58.204" maillog gives us the answer:
Code:
Jul 22 21:06:16 mail postfix/smtpd[1346789]: connect from unknown[211.216.58.204]
Jul 22 21:06:30 mail postfix/smtpd[1346789]: NOQUEUE: reject: RCPT from unknown[211.216.58.204]: 554 5.7.1 Service unavailable; Client host [211.216.58.204] blocked using bl.spamcop.net; Blocked - see https://www.spamcop.net/bl.shtml?211.216.58.204; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<1MKIMp>

This is the ‘postfix[mode=aggressive]’ in action as 'aggressive' means that fail2ban is considering failures from (among other signals):
  1. improper command pipelining
  2. Sender (or Recipient) address rejected
  3. authentication failed
  4. lost connection after... (DDoS)
  5. rBL check
So, the block of 211.216.58.204 at the end of the block list isn't coming from an auth failure, but from an rBL failure. And this fail2ban block means the banned host won't even be able to establish a connection until the ban has expired, much less try anything else and tie up system resources. And the 'cyano' address is entirely fictious, highlighting both a mix of real and imaginary credentials being tested. Interesting that it was attempting to send to a mail.ru address...
 
Back
Top