Resolved Postfix rate limiting

[Bobbbb]

Hello,

There are several big domains that frequently defer accepting mail from us causing long delays or rejections. Google, AOL, and Yahoo are examples. I'm considering trying the suggestions found in this online posting regarding rate limiting the sending of messages to those domains. In the below URL, please see the section titled "Different policies for different domains"...

http://steam.io/2013/04/01/postfix-rate-limiting/

Would these changes be safe to make on a CentOS 6.4 server running Plesk 11.0.9 with Postfix 2.8.4? Would any special modifications for Plesk be necessary?

Any thoughts or suggestions are welcome.

Regards,
Bob

 

[Bobbbb]

To save someone a click, here is the posting I was referring to.

Would these changes be safe to make on a CentOS 6.4 server running Plesk 11.0.9 with Postfix 2.8.4? Would any special modifications for Plesk be necessary?

Any thoughts or suggestions are welcome.

Postfix rate limiting – Politeness goes a long way
By Yuval on April 1, 2013 in Email, Postfix
When sending large amounts of bulk emails, it may be necessary to enforce rate limiting to prevent the ISP’s from blocking your servers.

Reputation
In the world of high volume senders, a key metric for detecting spammers is the IP addresses’s reputation. Building a good reputation for your servers takes time and hence patience but it pays off with both delivery rate and performance, allowing you to send more email from a single resource.
Until you have good reputation, and to gain a better one to start with, it is recommended that you enforce rate limiting on your MTA. In this post I will focus on Postfix rate limiting, which I’m guessing is why you are here anyway.

The dreaded 421 error
Most email service provers allows you to send them emails at a certain unpublished rate. Some are politer than others, such as lets say, Yahoo, but when sending to a list of more than about 1000 recipients, it more than possible that you will face a temporary block. To prevent it from happening you should fine tune your postfix policy to create some form of delay between the emails that are sent to the same domain.
If you already encountered this error, it may look like:

Our system has detected an unusual rate of 421-4.7.0 unsolicited mail originating from your IP address. To protect our 421-4.7.0 users from spam, mail sent from your IP address has been temporarily 421-4.7.0 blocked. Please visit http://www.google.com/mail/help/bulk_mail.html 421 4.7.0 to review our Bulk Email Senders Guidelines.

Postfix rate limiting configuration
My first example will be of a global configuration and assumes that your server is handling only outgoing emails.
We need to edit the main.cf file and add the following lines:

smtp_destination_concurrency_limit = 2
smtp_destination_rate_delay = 1s
smtp_extra_recipient_limit = 10
Lets go over each line:

• default_destination_concurrency_limit = 2
  This means that postfix will up to two concurrent connections per receiving domains.
  The default value is 20.
• default_destination_rate_delay = 1s
  Postfix will add a delay between each message to the same receiving domain. It overrides the previous rule and in this example, it will send one email after another with a delay of 1 second. If you want to disable this rule, either delete it or set to 0.
• default_extra_recipient_limit = 10
  Limit the number of recipients of each message. If a message had 20 recipients on the same domain, postfix will break it out to two different email messages instead of one.

Restart your postfix service and you are done.

If you are sending thousands of emails, and need to do so fairly quickly you will learn that this configuration will delay your rate considerably. Once you gain a better reputation, you will be able to move to the next step.

Different policies for different domains
Now, as you fine tune your policy, it may be apparent that some providers allow you to send emails at a higher rate, and some may need more delay between each message.
Lets create separate policies so we can optimise our server’s performance.

The first step will be to edit the master.cf file and add SMTP transports. Each transport will represent a policy for a group of receiving domain:

polite unix - - n - - smtp
turtle unix - - n - - smtp

Next, we need to map a domain to it’s transport name:
Edit the /etc/postfix/transport file and add the lines

gmail.com polite:
yahoo.com turtle:
hotmail.com polite:
Now we can add the policy to the master.cfmain.cf file:

transport_maps = hash:/etc/postfix/transport


polite_destination_concurrency_limit = 2
polite_destination_rate_delay = 0
polite_destination_recipient_limit = 5

turtle_destination_concurrency_limit = 1
turtle_destination_rate_delay = 3s
turtle_destination_recipient_limit = 2

Note that these values are just an example and you will have to tweak it yourself over time.


To activate these changes, we need to reload the postfix configuration:
postmap /etc/postfix/transport
service postfix restart


Read the ISP’s guidelines for bulk senders to better understand how to improve your reputation:

• Gmail – Best practices for bulk mail senders
• Yahoo – Bulk Senders Guidelines
• AOL – Bulk Sender Best Practices
• Hotmail – Troubleshooting
• A good reference book that I use – Postfix: The Definitive Guide
  

If you have any questions you can find me on IRC – #steamio on Freenode

 

[Bobbbb]

Just to be clear, the reason we're looking to limit the speed at which we sent messages to Google, and others, is NOTdue to excessive non-delivery notifications. Instead, it's primarily due to e-mail forwarding.

Let me explain...

We are a web hosting provider using Plesk11 on CentOS 6.4 with Postfix. Many of our customers, instead of checking mail directly from their accounts on our server using POP3 or IMAP, have set up e-mail forwarding to forward any e-mail sent to their domain's e-mail address to Gmail, Yahoo, Hotmail, or other large web based e-mail providers. I understand why customers prefer to do this. I also understand the problems that e-mail forwarding can cause.

So, in addition to the customers that do send and receive mail directly through their accounts on our server using POP3, IMAP, and SMTP, there are also a large number of e-mail addresses that our server receives mail for and then has to forward to their final destination (usually Gmail, AOL, Yahoo, Hotmail, etc.).

These e-mail addresses that are set up to forward mail, end up forwarding ALL mail, including spam. This causes Gmail, AOL, Yahoo, Hotmail, etc., to throttle/defer mail and causes long (days) delays, or bounces, for all e-mail users (not just the ones using forwarding).

We use several RBLs, in addition to MagicSpam for Plesk, to try to cut down on the amount of spam that reaches our user's mailboxes, as well as reduce the amount of spam that ends up being forwarded, but we still have trouble with being throttled by Google, etc.

That is why we would like to try to rate limit the mail going out to Google, etc., to avoid the large delays and bounces.

I hope I have explained well enough.

 

[IgorG]

I can say only, that we do not support this feature officially and never tested it. You can try to do it by yourself and share results. Thanks.

 

[Bobbbb]

It's been a few years. Here are the main.cf tweaks that I added and have been working well for the past several years:

smtpd_sasl_authenticated_header = yes
initial_destination_concurrency = 2
maximal_backoff_time = 8000s
default_destination_rate_delay = 12s
smtp_destination_concurrency_limit = 1
default_destination_recipient_limit = 10

 

[Jllynch]

Any idea how to do it per domain? The instructions at the top of the page don't work.

These are better than nothing though;

It's been a few years. Here are the main.cf tweaks that I added and have been working well for the past several years:

smtpd_sasl_authenticated_header = yes
initial_destination_concurrency = 2
maximal_backoff_time = 8000s
default_destination_rate_delay = 12s
smtp_destination_concurrency_limit = 1
default_destination_recipient_limit = 10

 

[Quintin]

Any idea how to do it per domain? The instructions at the top of the page don't work.

These are better than nothing though;

Holy thread resurrection batman!

Per domain I have had good results with postfwd. You can fine tune it and even do per email address. Additionally there is turtle where you can set recipient domains and make the postfix server be less aggressive with those.

Postfwd is powerful but a bit clunky to get configured, as it is basically a similar to firewall type setup for postfix, but instead of ip based rules, you have email address/domain based rules.

Below some examples I use:

id=YAHOO!; recipient_domain=~yahoo.*; action=rate(sender/1/300/REJECT only 1 messages per 5 minutes for $$recipient)
id=AOL!; recipient_domain=~aol.*; action=rate(sender/1/300/REJECT only 1 messages per 5 minutes for $$recipient)
id=ROUNDCUBE; client_address=127.0.0.1 ; action=rate(sender/5/300/REJECT only 2 messages per 1 minutes for $$sender)
id=ROUNDCUBE2; client_address=123.456.789.001 ; action=rate(sender/5/300/REJECT only 2 messages per 1 minutes for $$sender)
id=WEBDE; recipient_domain=~web.de; action=REJECT web.de is on a blocklist and will not be delivered

Hope this helps!