Upgrading from 12.0.18 -> 12.5.30 problems, plus fail2ban and ongoing DDoS!

[danami]

IPsets can help in that with Juggernaut Firewall you can enable dynamic blocklists (like TOR or Anon proxies), block entire countries or ASNs to proactively block bad guys very efficiently. If you really want to use fail2ban then I would recommend disabling some of the bad jails like the "apache-badbot". That thing is virtually useless and causes serious load with just a few domains.

 

[trialotto]

@danami,

You and me both know very well that ipsets can be used without Juggernaut: it simply is a simple package, that can be installed AND can also be used very easily (i.e. with some minor tweaks) in Fail2Ban.

Naturally, I agree that most of the default jails/actions/filters need some manual fine-tuning, but that is another topic.

However, whenever using Fail2Ban under high stress of attacks, the only thing that matters is the recidive jail. And even that can use some tweaking.

In general, you and me both know that any "true" DDoS cannot really be attacked effectively, not even by blocklists or IP blacklists.

In all aspects, dynamic blocking is a process with a learning curve and this simply means that any "true" DDoS (i.e. an attack using 1000s of IPs per second and revolving those IPs each time they have been used once or twice) will go faster than any dynamic blocking process can "learn bad IPs".

Moreover, what is in a name? Dynamic blocklists are in essence multiple static IP lists combined together, with the dynamics only resulting from the fact that the individual lists cause a small difference in the entire blocklist, when having a look from second to second.

In short, you are aware of all of the above and the above is not really relevant, just as the plugging is not really relevant (if you gain satisfied customers, I am happy for you and the customer, no worries there).

Personally, I find it problematic that it simply is "working towards a (paid or free) solution, that is not really a solution at all".

Instead, you could or should focus first on a solution that stabilizes the system (to allow other actions) and, afterwards, a solution that can prevent some issues re-occurring.

Your solution, being Juggernaut, is something that essentialy can help in the second phase of the process: prevention of problems (i.e. due to attacks) to re-occur.

Finally, all of the above is again with all due respect.

Regards....

 

[danami]

You and me both know very well that ipsets can be used without Juggernaut: it simply is a simple package, that can be installed AND can also be used very easily (i.e. with some minor tweaks) in Fail2Ban.

So whats your point? All of Plesk is based on free software. Sure I can configure Nginx, Apache and Postfix on my own to but Plesk putting everything together and making it easy to manage is where the value is. It saves me time. Its the same with Juggernaut Firewall.

In general, you and me both know that any "true" DDoS cannot really be attacked effectively, not even by blocklists or IP blacklists.

While I agree that a full on DDos does need dedicated hardware most of the attacks we see are usually just thousands of bots trying to brute force some service like SMTP_AUTH. If you want full real DDos protection then go with a provider like OVH (with hardware that costs big bucks).

In all aspects, dynamic blocking is a process with a learning curve and this simply means that any "true" DDoS (i.e. an attack using 1000s of IPs per second and revolving those IPs each time they have been used once or twice) will go faster than any dynamic blocking process can "learn bad IPs".

I've stopped SMTP_AUTH attacks with 40,000+ bots from 60+ countries easily using Juggernaut. Most of the compromised bots are from third world countries so blocking outgoing mail ports to those countries is trivial. Also if you manage many servers you can configure LFD to share blocks across all your servers. Its fun watching a botnet attack your honeypot while protecting all your other servers

We can agree to disagree

 

[trialotto]

@danami

This is remarkable and made me laugh:

If you want full real DDos protection then go with a provider like OVH (with hardware that costs big bucks).

Note that OVH is not the best hosting provider, a lot of network failures, a lot of wrong installations, buggy hardware and so on.

I am not sure what your argument in this is, but there are certainly better alternatives.

And the best alternative is one that most people do not think of: use a "catchbox", redirect all traffic to a machine that "traps" the attackers (and a simple tweak in the "catchbox" will get you all the bad IPs in a convenient list AND return some or thousands of error codes to the machines of the attackers, shutting them down completely).

But never mind.

I've stopped SMTP_AUTH attacks with 40,000+ bots from 60+ countries easily using Juggernaut. Most of the compromised bots are from third world countries so blocking outgoing mail ports to those countries is trivial. Also if you manage many servers you can configure LFD to share blocks across all your servers. Its fun watching a botnet attack your honeypot while protecting all your other servers

I am sure that you know what you are talking about, but

- SMTP related attacks are often the facade of some nasty game behind the screens: everyone expects SMTP attacks and is prepared to that or busy fighting those attacks, but if you have a close look at traffic, most damage is already done by another attack behind the screens,

- 40000+ bots from 60+ countries? Ehm, those are often 2 or 3 countries, with a 1 to 100 "hack teams" per country (including governmental teams), staging a huge number of re-routed attacks of the MITM kind, trying to shield off the origin of the attack. In essence, knowing the original bad IPs (those are limited to less than 1000, they tend to be rather fixed) AND using a proper defense to MITM would or should not lead to the numbers you stated. I am bit surprised, you can imagine.

- LFD as in "part of CSF"? Really, ConfigServer is not that nice. For one reason, it does not interact nicely with Fail2Ban, a default package of Plesk. I am putting it mildly.

- "block sharing", as in sharing resources between servers? Well yes, very nice, but this can be executed in Fail2Ban (piece of cake, but the wrong programme to do so) and there is always the default option to have Fail2Ban make use or report to "dynamic" (what is in a name) blocklists.

We can agree to disagree

No, we can not.

We can agree that the default Fail2Ban (as provided with Plesk or even in general) needs some tweaking, but has a lot of functionality.

We can also agree that we can point forum members to the pitfallls of Fail2Ban and that a (genuine) firewall is better (prevention of attacks, not actions afterwards).

We can also agree that there are a lot of solutions, most of them being a "reinvention of the wheel".

We can finally agree that some cooperation, i.e. sharing of bad IP blocks, should be present..........that is the most important part.

After all, every single box running Fail2Ban, but not sharing information does imply that other boxes are getting attacked.

Again, attacks are often originating from a small number of teams with a small number of servers: they only use many HACKED servers to gain momentum and create attacks of the MITM kind, which kind of attacks use many, many servers (with those servers being the hacked ones).

In short, if everybody shares some knowledge and the information about bad IPs, then there would be a large reduction in attacks.

Come to think of it, now I recall why I thought your plug for Juggernaut to be a little bit shameful: it is selling a product, not sharing knowledge or information.


By the way, I want this discussion to be neutral and valuable for the remainder of the forum members.

We can discuss a lot, but let´s not do that, just provide the information that can help forum members shield off their systems, from attacks.

Regards.....

 

[danami]

@trialotto Someone is having a bad day

@danami
Note that OVH is not the best hosting provider, a lot of network failures, a lot of wrong installations, buggy hardware and so on.

I agree that OVH does have their issues but they are one of the few providers that I know that offer real DDoS protection on their VM's and dedicated servers at such a low price point. Just make sure to turn it on through their control panel way before hand. It needs to learn your traffic to your server.

@danami
In short, if everybody shares some knowledge and the information about bad IPs, then there would be a large reduction in attacks.

That's exactly what we do. We use the iplists from the Firehol project and contribute back to it. Costa Tsaousis is doing an amazing job. I suggest you take a look at them. http://iplists.firehol.org/

@danami
Come to think of it, now I recall why I thought your plug for Juggernaut to be a little bit shameful: it is selling a product, not sharing knowledge or information.

1. I contribute Plesk related patches patches and donate money to the CSF project on a regular basis. You can install and run CSF with no Juggernaut interface if you want. It's completely free. I've also given away all my Firehol work to CSF which you can use for free. See: http://forum.configserver.com/viewtopic.php?f=5&t=9024

2. I've also reported Plesk security vulnerabilities privately to the Plesk team making Plesk more secure.

Oh well I guess you can't please everyone. Cheers!

 

[trialotto]

@danami,

Do you have a bad day then?

OVH is quite expensive and the tools you are referring to (i.e. DDoS protection) are nothing else that hardware and/or software based solutions. Nothing new there.

It is not a "good or outstanding" feature, all or most (large) hosting providers have the same tools, but they do not boast about it: it is simply "on" automatically, as part of the network infrastructure and the implementation of "common and good practice".

Again, never mind.

The firehol project is interesting, but a little bit strange and a re-invention of the wheel: for instance, cisco based lists are better and more elaborate, just other commercial lists (note: there are various very interesting paid-for solutions, with huge potential. I will not bother to mention them, they are really costly).

The advantage of this project is the sharing of knowledge and information, the disadvantage is that the list is as good as the "people contributing".

To compare, the "OVH solution" is associated with a lot of false positives (almost any IP passes) and the "Firehol solution" (or any other IP blacklist solution) is associated with a lot of false negatives (too much IPs are banned, including the genuine IPs).

By the way, a rare type of attack (usually used to perform an attack on huge companies) is the following: start a simple attack (in order to learn about the "defense policies" of a set of servers), create a DDoS (to weaken the system and often monitor admin traffic, since they have to intervene) AND add the most important IPs (that are crucial to maintain a set of servers) to a blacklist. The blacklist addition often results in the sysadmins not being able to access the system remotely, limiting access to server and rack spaces in the datacenter. That takes time and/or there is always the weak link in the infrastructure that can be attacked and entered (when the system is almost fully down). And there it is, the backport to the system, but it does not end there. The hackers will often bring the system online again, as if nothing has happened. There you have it, an accesspoint to have a peek.

So, in short, blacklists are not always your best friend.

Another reason for that is that most blacklist do allow traffic from the contributor´s servers: a type of whitelisting, also used by hackers.

And there are the couple of blacklists that are actually used to whitelist bad IPs.

Note that I do not write this for you, but for the general forum member, interested in the concept of blacklists and the advantages/disadvantages thereof.

Personally, I find firewalls, fail2ban, blacklists and similar tools to be overrated for 2 reasons:

1) just use a proxy as a first line of defense, with huge capacity and possibilities,

2) just spend time to write scripts (to be run at the proxy end) that actually check for BAD CODE (and not bad IPs)

and that actually is what large providers (microsoft, amazon, google) do.

With respect to:

1. I contribute Plesk related patches patches and donate money to the CSF project on a regular basis. You can install and run CSF with no Juggernaut interface if you want. It's completely free. I've also given away all my Firehol work to CSF which you can use for free.

I can only say: nice to hear.

Again, CSF does not work nicely with Plesk (and CSF has some bug and vulnerability issues).

With respect to:

2. I've also reported Plesk security vulnerabilities privately to the Plesk team making Plesk more secure.

I can mention that I have seen some of them and I must admit that they are sometimes actually adding value.

Problem is that not everything is patched according to your recommendations, the same applies if I make a recommendation or request.

Just accept that Plesk is just as safe or just as vulnerable as any other standard package on the market.

Most security fixes are added to the package itself, by the package author and contributors, after which Plesk uses the latest release of the fixed package.

That is the way it works in the linux world: it can take weeks, months and even years before known security issues are fixed.


You know what, I have to say the following: in the more than ten years of running a lot of servers (note the understatement), we have never had issues with security.

If you ask me, the reason for that is that we have never used any fancy stuff/tools AND we never make a fuss about the hot topic of security.

Really, the basic rule is: do not increase the attack surface (and more tools means more attack surface, also applies to tools that are intended to increase security).


Anyway, thanks for the (implicit) tip that Juggernaut is essentially a version of CSF, which can be obtained for free or a small donation.

I certainly value and appreciate your honesty, this type of sharing information is valuable for forum members.

If someone wants to use something like CSF, one can first test for free with CSF and, afterwards, try the (more expensive) Juggernaut (that is more user friendly).

Regards....