Forwarded to devs Better use robots meta tag instead of robots.txt to hide Plesk

[King555]

Username:

TITLE

Better use robots meta tag instead of robots.txt to hide Plesk

PRODUCT, VERSION, OPERATING SYSTEM, ARCHITECTURE

Plesk Obsidian 18.0.81 Update #3 Web Admin Edition, Ubuntu 24.04.5 LTS

PROBLEM DESCRIPTION

This is not really a bug, but a technical design flaw. I'm pretty sure that having a robots.txt file with the content

User-agent: *
Disallow: /

will not prevent Google and other crawlers from indexing the page itself (or the domain name), but only the content of the page.

Which means that the URL of my Plesk installation might be revealed to the public via Google.

That's why I would like to suggest to use the robots meta tag on the page instead.

The file I mean is this:
/opt/psa/admin/htdocs/robots.txt

There are also these files, but I guess they are part of the third-party tools and are not directly reachable anyway:
/opt/psa/phpMyAdmin/robots.txt
/usr/share/grafana/public/robots.txt

STEPS TO REPRODUCE

See robots.txt in root folder and HTML code of login page of Plesk.

ACTUAL RESULT

robots.txt blocks everything, HTML page does not prevent actual indexing.

EXPECTED RESULT

Empty robots.txt file, this added to HTML code of Plesk (at least to the login page):

<meta name="robots" content="noindex, nofollow">

ANY ADDITIONAL INFORMATION

(DID NOT ANSWER QUESTION)

YOUR EXPECTATIONS FROM PLESK SERVICE TEAM

Confirm bug

 

[Sebahat.hadzhi]

Thank you for the report, @King555 . I believe the current robots.txt behavior is a deliberate rather than a defect. It also isn't a security issue, since hiding the panel from search engines doesn't replace authentication and other access controls. I can't classify this as a bug. Nevertheless, but I opened an internal case for our team to confirm if there's ground for improvement in this regard. I will follow up as soon as I get an update.

 

[Sebahat.hadzhi]

@King555 , a case for the improvement of the behavior was registered (PPPM-15585). At this point, I cannot provide an ETA on when the change will be released. Until then, the proposed workaround is to add add_header X-Robots-Tag "noindex, nofollow" always; to the server block in /etc/sw-cp-server/conf.d/plesk.conf (per How to add headers to Plesk Web Interface pages? ), serve a permissive robots.txt, and reload sw-cp-server. plesk.conf is regenerated on updates, so the change must be reapplied as per https://support.plesk.com/hc/en-us/...k-conf-and-avoid-overwriting-by-Plesk-updates.
Thank you for bringing our attention to the matter.

 

[King555]

Thank you, this is good news!

As an additional note: I personally use the following full list to make it very clear, although the above is usually sufficient:

noindex, nofollow, noarchive, nosnippet, noimageindex, notranslate

 

[10xlinks]

One thing worth adding is that noindex only works if crawlers are actually allowed to fetch the page and see it, so the old Disallow rule in robots.txt can quietly work against you. Once you switch to the header workaround, run a quick curl -I against the panel URL on port 8443 and confirm the X-Robots-Tag line really shows up. If the hostname already got indexed, a removal request in Search Console for that host speeds things along. Also keep in mind that the panel hostname can leak through certificate transparency logs and old DNS records, so a firewall allowlist for admin IPs is the more reliable way to keep strangers away than anything search related.

 

[King555]

noindex only works if crawlers are actually allowed to fetch the page

Exactly. That's why I created this thread.

Also keep in mind that the panel hostname can leak through certificate transparency logs and old DNS records

I know that this does not really improve security, but preventing a listing in common search engines is a first step (a small one, but a step).