@Xavier12
First of all, I am rather suprised that you choose the Cloudlinux system, without having lvemanager enabled. In such a situation, you would be better off with Ubuntu.
Nevertheless, some general remarks and tips and for the sake of convenience, I will quote some parts of your response.
We are a little concerned about enabling the Plesk firewall due to previously bad experience. After we enabled it in the past, many ports were blocked/modified and several services somehow stopped working or functioning correct. Because of this previous issue, we ended up having to re-install a whole new server and plesk and refrain from using the Plesk firewall option.
You should not worry about that.
In general, the Plesk firewall is a GUI for iptables management and it certainly will not cause specific services to stop working and/or working improper.
It can be the case that you blocked some ports, not allowing specific services to connect through the firewall.
If you want to test Plesk firewall, just enable it with all rulesets set on "allow", implying that all traffic is getting through AND also implying that, in the very rare case that your system stops working properly, it indeed was the Plesk firewall causing your system to stop functioning.
I am pretty sure that you can enable the Plesk firewall without any problems for the CloudLinux system.
Make sure that you deactivate Fail2Ban service before you enable the Plesk Firewall test, otherwise your "test results" would be biased.
Since we have a good amount of customers, it would be difficult to create a robots.txt files for each website installation.
Not really.
In essence, the robots.txt can be identical on all sites, certainly if the purpose is to block specific (bad) crawlers.
The one (and only) robots.txt file can be copied to each of the httpdocs directories for (all) the domains you have. That seems to be a lot of work, but it really isn´t.
On the other hand, why bother if one has Fail2Ban?
Ehm, there we are again: we want Fail2Ban, but Fail2Ban has some peculiar issues.
Certainly, you are right when stating
There is an option to permanently ban the unwanted ip addresses by sending them over to iptables permanently, but fail2ban performance just isnt performing as it should.
and I must say that the bold part of the quote always has been true, in the sense that not every function in Fail2Ban functions properly.
Nevertheless, most of the "alleged Fail2Ban issues" have something to do with bad configuration.
And, more important, configuration improvement can also be used to "work-around" the problems with specific Fail2Ban functions.
Problem is..... while writing this post, I became a little bit unfocused, since I thought of some specific Fail2Ban setup that we tested in a far past.
That specific setup can be a full resolution of your (and other´s) problem with Fail2Ban.
I will first have a look at that, is that a good suggestion?
Regards....