Issue ACL setup for Google website crawl

[Thomas Oryon]

Server operating system version

Redhat linux

Plesk version and microupdate number

18.0.74

Hello Support,

We would like to setup ACL (Access Control Limit) to stop/block the Google to crawl the website. Also, once we setup this option, the domain which was already added and upcoming domain add will also should not crawl by Google. Is that possible to setup? If yes, please let us know the steps to setup this.

Awaiting your reply.

 

[scsa20]

Why not just configure robots.txt? Google does honors whatever is configured on robots.txt. You can learn more over at https://www.cloudflare.com/learning/bots/what-is-robots-txt/ and you can utilize Robots.txt Generator

 

[Thomas Oryon]

Hello Support,

We can't add robots.txt for each domain name. We wish to block server wide. Can you let us know how to setup this?

Awaiting your reply

 

[scsa20]

You're best bet would still use robots.txt unless you want to edit your firewall rules to block all of Googlebot's IP addresses. See Verify Requests from Google Crawlers and Fetchers | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers for list.

 

[Thomas Oryon]

Hello Support,

Thanks for the update. Can you let us know the steps to enable the robots.txt in server wide?

Awaiting your reply.

 

[scsa20]

robots.txt cannot be set server wide, it needs to be done per domain, otherwise just block the IP address at the firewall level (check the link I provided originally to get the list of IP addresses used by the crawlers).