Googlebots & Google Ratings. Fail on Plesk 12

[Tsi-Shawn]

I have two linux servers. Both CentOS 6.5. One ran Plesk 11..0.9 with over a hundred sites on it. I migrated another 120+ then upgraded to Plesk 12.

The other server was created from scratch with Plesk 12 on it. Last week I migrated 300 domains over to it.

On server 1 only the domains that were created on it pre plesk 12 are still able to be indexed by googlebots.
On server 2 none of the domains can.

Also all the domains that were migrated onto both servers cannot use Google Insights (Checks for page speeds etc) and have dropped off Google listing. All bad and generating a ton of calls from unhappy customers.

I do ran iptables and fail2ban but the iptables have not changed since the migration and fail2ban has only dropped a few badbots and of them they are not google IP's.

I first thought it might be nginx so I turned that off but all of the Google problems remain.

Anyone run into this or have a fix for this? I need this resolved asap and I would rather not call it in and open a ticket.

 

[Tsi-Shawn]

Sure enough. I removed my custom blacklists from my iptables and Google Insights is now working. What does this say of google? All of my custom blacklist DROPS come from logwatch and are brute force attempts to root. tsk tsk google.

 

[UFHH01]

... that's why fail2ban has the option to set some trusted IPs ... you have as well the option to define IP-ranges. I use all Google IP-ranges, the ones from Yahoo, Microsoft and some other trustfull spiders and hosts on my servers

 

[Tsi-Shawn]

If you have a list of Google IP's could you pm them to me? I looked around and get from Google that there don't have a "set" list.

 

[UFHH01]

Well, for Google it is quite easy:

http://bgp.he.net/irr/as-set/AS-GOOGLE​

Now you see the Admin-C and Tech-C which is "AS15169" and afterwards you take this no. 15169 and look up at:

https://www.dan.me.uk/bgplookup?asn=15169​

Voilá. ^^

 

[Tsi-Shawn]

Awesome thanks for this. Didn't even think to look here.