Issue 403 GET /sitemap_index.xml HTTP/1.0 help

[ryuseei]

When I add the sitemap to the search console, it gives 403 error. I’m looking at the logs on the Plesk and I’m getting a 403 error. I am using the sitemap feature of rank math. When you physically create a sitemap.xml file, put it in htdoc and edit from robots.txt, you can see it, but you know rank math does not give a physical xml file, so I edit from robots.txt only but I get a 403 error. I set up a second site on the same server and it worked without any problems. Something is blocking googlebot and bingbot but I couldn’t figure it out. The screenshot below is the log result when I query sitemap_index.xml in the search console. Mod_security fail2ban firewall vs. I turned it off, disabled all plugins, still the same.

 

[scsa20]

So did a quick test, when you go to yourdomain.tld/sitemap_index.xml it should still load something, if it doesn't then more likely you do not have rewritten URLs configured correctly. How you configure it is in one of 2 ways depending on which web server you're using. If you're using Apache using the generic .htaccess that's provided by wordpress will work and should just work. If you're using NGINX, you'll need to make sure you've included URL Rewrite rules in the additional NGINX directive.

 

[ryuseei]

So did a quick test, when you go to yourdomain.tld/sitemap_index.xml it should still load something, if it doesn't then more likely you do not have rewritten URLs configured correctly. How you configure it is in one of 2 ways depending on which web server you're using. If you're using Apache using the generic .htaccess that's provided by wordpress will work and should just work. If you're using NGINX, you'll need to make sure you've included URL Rewrite rules in the additional NGINX directive.

When I type domain.com/sitemap_index.xml, the sitemap appears. There is no problem here. The problem is googlebot cannot read sitemap.
I think the problem is here.


Hugo's URL Viewer When I search from this site, it says text/html, but it should be text/xml. Googlebot cannot read it because it is html. I just don't know how to fix this problem.

 

[scsa20]

I'm not able to replicate the issue. Google accepted the site map with no issues and Hugo's URL Viewer is showing up as text/xml to me. I don't know what your current setup is like, or your environment for that matter, but seems to be a weird server config on your end if it's spitting out text/html instead of text/xml. You can try going into the settings of rankmath and turning off HTML site and see if that helps.

 

[ryuseei]

I'm not able to replicate the issue. Google accepted the site map with no issues and Hugo's URL Viewer is showing up as text/xml to me. I don't know what your current setup is like, or your environment for that matter, but seems to be a weird server config on your end if it's spitting out text/html instead of text/xml. You can try going into the settings of rankmath and turning off HTML site and see if that helps.

I have the same problem with rank math yoast and all other plugins. I reached out to Plesk support and they told me it could be caused by wptoolkit bot protection. I can't turn it off right now because I left this feature turned on in the trial version. I have to buy it to close (This must be a joke)

My theory is that the bot comes to the url to read the file but bot protection or anything in wptoolkit redirects the bot to a different location or shows the wrong page. Not only googlebot but bingbot and all other bots have this problem. None are working.

 

[scsa20]

I have everything enabled including bot protection and it still worked fine for me:




As for your reverting, you should still be able to but what you showed in the screen shot won't work because you selected a security item that isn't already protected. You need to put a checkmark for the one you don't want the protection from.

 

[ryuseei]

I have everything enabled including bot protection and it still worked fine for me:




As for your reverting, you should still be able to but what you showed in the screen shot won't work because you selected a security item that isn't already protected. You need to put a checkmark for the one you don't want the protection from.

Yes, I still haven't been able to fix the problem. I don't know why I'm getting this error everything seems correct. I don't think I can solve this problem. I've been struggling for days.

 

[Peter Debik]

wptoolkit bot protection. I can't turn it off right now because I left this feature turned on in the trial version. I have to buy it to close (This must be a joke)

@ryuseei It's a trial version and it is clear when that trial expires from the beginning. Why do you expect that you'll be able to continue to use the full feature set after the trial period has expired? A trial version is not meant for production.

You can try these workarounds to remove bot protection from the web server configuration files:

A) Clone to a temporary domain and back

1) Create a temporary domain, e.g. staging.yourdomain.tld

2) Clone or copy your Wordpress installation to that domain.

3) Remove it completely from your source domain.

4) Clone or copy it from your temporary domain to your previous source domain.

or

B) Detach website from WP Toolkit and reconfigure the web server configuration files.

1) Detach website from WP Toolkit


2) Remove .wp-toolkit-ignore file from the document root directory of your Wordpress installation.

3) Go to Tools & Settings > Assistance & Troubleshooting > Webserver Configurations Troubleshooter.

4) Select your domain including the subdomain entries from the filter, then run "Rebuild" > "Selected".