Issue Google crawler cannot crawl my page sitewide issue error connecting to server

[palletdeal]

Server operating system version

Ubuntu 20.04 x64

Plesk version and microupdate number

18.0.48

Hi all,
I've been trying for a few months to get my sitemap updated in search console, however Google cant crawl my pages due to an server connection error.

Does somebody maybe know how to resolve this issue?

Kind regards,

Marc

 

[Peter Debik]

Make sure that Googlebot is not blocked by
- Additional Nginx directives
- Additional Apache directives
- A bot protection plugin that may exist in your website
- .htaccess rules
- Fail2Ban
A list of GoogleBot IP addresses is available here: https://developers.google.com/search/apis/ipranges/googlebot.json

 

[palletdeal]

Hi Peter,
Thank you for your reply. I have solved the issue by disabling IP address banning.
I don't know if its necessary to have it activated, otherwise I should add all of Google's IP adresses.

 

[Peter Debik]

Solution may not be easy in this case, at least require some work. If Google is blocked, then very likely this is due to a directory or file in a website of yours that is known to Google or used by the website, but is password-protected or protected an .htaccess rule. When the bot hits that file, it creates an entry in the error log "access denied". This will be seen by Fail2Ban so that Fail2ban treats the visit as if a real intruder tried to access protected directories or files.

You will need to check the website logs for "access denied" entries to find the root cause.