Dictionary Attack Crippling Us

[genxer]

Anyone out there know how to stop this. I have a PSA 7.1 system. We are getting mail @ a rate of 2.4 mililion (yes million per hour)

I've used IP tables ot block IP's however we are also blocking valid people, which makes a whole nother set of problems.

I've asked plesk for help and I get the standard , up grade mantra, but they fail to recognize that it takes better than 12 hours for our server to upgrade and JUST upgraded to 7.1 at the beginning of december.

I don't know what to do but any much longer we might as well commit me to the mental hospital .

Please help...any body please.

 

[webrebel]

dam...you do have relaying OFF don't you???? Then turn off any email catchalls.

Short of that I'd be all over my datacenter to rescue me! Sounds like something else is going on here...dos attack etc

 

[genxer]

Yup, relaying is off. we are getting emails for a@ domain, aa@, aaron@ [email protected]@...zeus@

I've black holed all the catchalls by rewriting the .qmail-default to a mailbox that is /dev/null AND removed oviedair from it. So I've now got the severs blocked from 1/2 the world except the world is now calling me complalining because they can't send mail.

Before we though up the firewalls. the connects where huge even with the /dev/null's we were getting killed.

I just don't know what to do.

 

[Cranky]

Are these all incoming to [email protected]? If so, why not just remove the MX record?

 

[genxer]

Interesting Idea to remove the MX records just to those domains however, customers do receive email @ their domain so I would think they wouldn't get the legit email then.

But i am glad to see some folks thinking. How can we keep the good and ditch the bad. Now one of my customers said only accept a few emails at a time per address, but I have no idea how to set that parameter up. Like if a domain recieves more than 5 emails in a 1-2 minute timeframe ditch the rest, but I dont' know how to set that up either.

I did call UUnet and they said nothing they can do but stop all port 25 connections which means no one sends email to any of our servers....that would be bad.

 

[Cranky]

I'd say that if you're receiving as many emails as you say you're receiving then you need to remove that domain's mail service from your server to keep other customers online. Wouldn't you agree? If there's no MX record for the domain the attacker will likely give up pretty soon then you can try restoring later. Regardless though, if you're receiving anywhere near as many emails as you think you are the customer isn't likely to find much use in legitimate email.

Actually, can't you just disable catch-all (set it to the REJECT option so mail doesn't enter your mailqueue)?

 

[genxer]

I've asked Plesk to add that for the 7.1 version where as our servers are too large to just 'upgrade without hours/days of downtime.

I was hoping there was something we could add to qmail control or some way to limit what would be accepted to help with the problem.

Catchalls have all been eliimited via a black hole email. If there is another way outside psa to handle this until they can patch it, I"m open to any suggestions.

The people that are being attacked do receive legit email where as all of the attacked sites site on top of google and get 100,000 + visitors, which is how I think their domains have gotten tagged in the first place, funny thing is these aren't large companies but 2-3 employees sites.

:::sigh::: spammers should be shot

 

[Ales]

Just as a sidenote - like in every argument, it usually takes two... We've had a few DDOS experiences in the past and after we researched the matter, the target often knew who the attacker was.

There was a lot of "well, now that you mention, there was this guy that was pissed at us..." etc. etc. Sadly to say, on couple of occasions our own customers were pretty reluctant to volunteer the information because they realised that they caused the attack... You know, it's easy to find two %&"%& on the net, one with a big mouth and another with resources and knowledge about DDOS.

I'm not saying that this is the case here but anyway, there might be a way to stop this by finding out who might be targeting your customer and eliminating the cause. If not, I'd advise you to explain to your customers exactly what is happening and let them choose among the possibilities (remove the MX record, point it to a dedicated server they rent for the time being, etc.). If they don't want to work with you, do whatever you think is best.

Sadly, the attacker has the advantage here... It's hard to fight a determined DDOS attack and if this is indeed a DDOS targeted at a specific domain, mail could be only the beginning...

 

[genxer]

thanks, I dont think this is anything more than they sending all these random emails to the sites sitting on the top of google. Now that might be part of it. But we blocked quite a bit of europe and asia and although I have international customers calling and complaiing all over the place, it has stemmed it some. Thing is where i blocked, its highly unlikely its a person selling mexican food or baby close to red hot asian babes ...blah blah

I just don't know what to do. I am reading that the whole problem is the way plesk has complied qmail and it can't just be recompiled without them. Was hooping some of you know how to work around that.

 

[genxer]

Interesting Idea to remove the MX records just to those domains however, customers do receive email @ their domain so I would think they wouldn't get the legit email then.

But i am glad to see some folks thinking. How can we keep the good and ditch the bad. Now one of my customers said only accept a few emails at a time per address, but I have no idea how to set that parameter up. Like if a domain recieves more than 5 emails in a 1-2 minute timeframe ditch the rest, but I dont' know how to set that up either.

I did call UUnet and they said nothing they can do but stop all port 25 connections which means no one sends email to any of our servers....that would be bad.

 

[todrules]

I just had the same problem a few weeks ago. It was crippling our server, and actually for a few hours, we had to call our backbone provider and block all port 25 connections.

I was trying some of the same things you are. However, there are a couple of things that I found that really worked. One of them is tarpitting. This doesn't actually block an IP or domain, but if somebody is sending you an email with alot (like 10K) recipients, then you would add a delay for each recipient. The spammer would then give up because it would take too long to send you any email.

Another thing to use is a RBL like spamcop. This is a GREAT service, and they will keep your blacklist up to date and more accurate then you could alone.

As for removing the MX records, it might work, but if you're that desparate, just make the MX record for the targeted domain point to 127.0.0.1. This makes the spammer deliver all the mail to their own server and bogs them down so they cant spam anybody.

 

[qnet]

broken qmail is problem

The fact that qmail does not issue a 5xx response to invalid mailboxes is the cause of most of your problems. It's unfortunate that the qmail designer has rectal-cranial inversion when it comes to this particular subject. It's also very unfortunate that Plesk hasn't done anything to correct this serious problem.

I've created a patch for my PSA 6 system that returns 5xx responses during RCPT TO if the mailbox (or alias) doesn't exist on the server. I'd be happy to send this patch to Plesk for incorporation into a future release. The patch creates a new qmail-smtpd and adds qmail-verify. It also adds reasonable syslog information (like the IP address of the connecting host). Qmail logs basically ****.

If you're using anything like a Barracuda with per-user quarantines, you need this patch or you'll end up with thousands of quarantines for non-existant mailboxes.

 

[voodoochile]

The default behavior of a plesk domain is to bounce the message with a "this user doesn't exist" message. I'd STRONGLY suggest shutting this off for all of your domains as in the event of a dictionary attack, your PSA box respondes to each mail with a mail of it's own stating "user not here". This effectivly doubles your mailq and that's bad. =)

If you go into the domain admin portion of PSA, then mail, you should be able to tell it to "reject" as opposed to bounce. (at least under 7.5.x), if you've got a ton of domains on your box I can give you a sql query I used to update all domains to reject.

Chris

 

[Foxxx]

Whats that querie?

Please let me know..

 

[voodoochile]

Here ya go, sorry for the delay.

update Parameters set value ="reject" where value like '%bounce%';

 

[Spyder]

We use 4 metods to try reduce this kind of problem (attacks and SPAM)

1. @ our real firewall we set somthing like "max tcp connections from any source" in this way some attacker/spamer can only open 5/10/20/n tcp connections from any IP to Any internal host.

2. @ our real firewall again we can setup one smtp proxy where we can force every smtp connection to pass under filters, rbls and etc (this is not in use anymore because we adopted the next resource)

3. Buyed one barracudanetworks apliance... and all messages pass in this apliance .. and believe this works great ..

4. Setup 4PSA (spamd and av ) in servers...

5. setup SMAP (from fwtk) but don´t setup this in plesk only in old servers.

the cronological order is 5, 1, 2, 4, 3 (we working ith inet since 1995) and some methods are disabled now.

Besides the disable MX (or put this to loopback ) if ou can try reduce the max tcp connections from some hosts to your hosts.

 

[Foxxx]

Hi,

Thank you for your answer. I changed everything in a few seconds now.

Kind regards,
Foxxx

 

[superbock]

Attention:

If no "MX" record is found, the domain's direct "A" record will be used... This is standard SMTP protocol procedure. Better blackhole it than remove it. But some spammers also delivery directly on the A record, disregarding MX's...

 

[stupidnic]

This is an inherent problem with qmail. Qmail accepts anything for a domain that is in its rcthosts/virtualdomains file. Once qmail has determined that the mail message isn't for a valid user, it bounces the message and sends a non-delivery report (or bounce message). So for every one of these dictionary attacks, you are actually dealing with two messages (the original which bounces, and the subsequent non-delivery report). Since spammers never (or almost never use valid addresses... certainly nothing that would come to them) these messages either fill up the outbound queue or result in double bounces.

It is possible to mitigate some of this with proper adjustment of qmail settings (doublebounceto, and catchalls that go to /dev/null).

I have found that the best method is greylisting. Greylisting takes the sending IP address, the from address, and the to address and creates a unique identifier for that message/sender. The first time it sees a new identifier that it has not seen before, the server will bounce the message with a soft bounce (deferral notice 4xx smtp message) and start a timer (typically 5 mintues). Here is where the real action occurs. The SMTP specification says that a mail server should keep retrying to deliver a message to the server at set intervals. Legitmate mail sent through "proper" mail servers will keep retrying the message, and once the timer has expired for that message identifier it will allow the message through. The next time the IP/from/To combo is seen by the server the message will not be delayed, it will be delivered right away, and so long as at least one message from that IP/From/To combo is seen with in a defined interval of days (normally 45 days), the interval of days is reset back to the maximum and starts counting downward again.

How does this effect spammers? Well, most spamming software is all about number of messages delivered per minute/hour/day. As such most spamming software is optimized NOT to follow the SMTP specification, so the first attempt a spammer makes, will result in a bounce and the spammer software will make a note that the email address isn't valid, and move on to the next delivery. Because spammers like to prune their lists (makes them more valuable and faster) they will typically remove addresses that bounce. This will help lighten the amount of spam that you receive.

I currently offer this service to a couple of clients that were getting a vast amount of dictionary based spam (<anything>[email protected]) to the tune of over 1 million messages daily. Since we have implemented greylisting, the volume has drop significantly.

How do you do it?

Setup a second server with just postfix on it (I personally like postfix over qmail... and I use to be a qmail diehard... take that for what it is worth). Setup the domains that it will be receiving mail for, and set those domains to forward to your mail servers IP (like smtproutes in qmail). Then set the MX record for the domains that are the problem, and point them to the postfix server.

Then flip the switch and stand back. It will take about 24 hours for the old MX records to expire and the new MX records to be picked up by the spammers.

This solution offers the best of both worlds, as one the bounce occurs VERY early in the SMTP process, so the load is very low as well as the bandwidth, as the SMTP conversation never gets to the DATA portion of the transfer so that should help out a lot. Also those customers on those domains will get their mail still. Plus... they will get less spam as well...

If you wanted to do it all on one server, you would need to setup postfix to run on port 25, and have qmail run on another port on localhost and have postfix hand off to the localhost and port you have qmail running on. It isn't that hard, just takes some doing to get everything setup properly. This of course would require a little more maintenance to ensure that all domains that are added in Plesk are also added to postfixs configuration to allow delivery.

If you really want to wow them, setup some really cool MRTG graphs.

http://www.spambuffer.com/mrtg/postfix.html

Hope that helps,

Tom
aka: stupidnic

 

[voodoochile]

My only problem with the above is that bounce behaviour on large servers is bad. You're not really solving any problems, just offloading it onto another server, which adds overhead to your operation and in turn extra costs, not to mention added layers of "things that can go wrong when rejecting the message altogether solves the problem at the frontline, rather than thowing servers, time and support at a spam problem, just stop it before it gets "expensive".

Another issue is that you assume that since a spammer gets bouncebacks from a server saying [email protected] isn't a valid user, that in turn, that account will never get spammed again. This is wrong as there's a ton of spammers out there, and they dont' really care too much about maintaining lists (hence why the use dictionary attacks in the first place). They just hammer out as much email as possible and dont' really worry about the bounces and whatnot. Hell, most of them forge the headers anyway so they never get the bouncebacks.

This in turn can also open you up to a different (and just as abused) method of spamming using bounces. When they send the email to a domain they KNOW bounces, they forge the from: portion of the header to be the real recpient of the spam, so your server does a dandy job of relaying the spam where it was supposed to go anyway and helped them cover their tracks.