Virtual robots.txt in wordpress toolkit...where is it?

[WhatWouldJesusDo]

The default setting is the following...

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

First of all, there is NO documentation on this file ANYWHERE.
Where is it?
How do you change it or disable it?
Why does it replace the robots.txt file I already have?
Any answers would much appreciated. Thank you.

 

[virtubox]

The default setting is the following...

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

First of all, there is NO documentation on this file ANYWHERE.
Where is it?
How do you change it or disable it?
Why does it replace the robots.txt file I already have?
Any answers would much appreciated. Thank you.

Hello,

robots.txt file is located in your website root folder (probably httpdocs) . But this file isn't managed by WordPress toolkit, and will probably never be managed by WP toolkit.
The content you have posted previously is most of time defined by the plugin Yoast SEO. Not by Plesk.

 

[WhatWouldJesusDo]

Maybe I should have been clearer...I already created a great robots.txt in my root folder, and when I activated the wordpress toolkit; it overtook it and I can not disable this virtual robots.txt file because it is not available to do. You need to do some research, because YES there is a virtual robots.txt created by plesk wordpress toolkit. Google it and you will see. So is there anyone in here with ACTUAL experience with this instance and does anyone know how to disable it?

 

[IgorG]

is there anyone in here with ACTUAL experience

From my own experience.
I have installed WP to my domain with WP Toolkit. After a while, I noticed a complaint that robot.txt file was missing, it seems from Google Analytics. I went to the root directory of my site and created my own robot.txt file there. It's all. Now there are no problems, everything works as it is expected.

 

[WhatWouldJesusDo]

I have a robots.txt file in my root. It is NOT what shows when I go to check it (yoursite)/robots.txt; it shows the virtual file installed or activated by Wordpress toolkit.

 

[virtubox]

I have a robots.txt file in my root. It is NOT what shows when I go to check it (yoursite)/robots.txt; it shows the virtual file installed or activated by Wordpress toolkit.

When I update my robots.txt file, it's also updated when I browse mywebsite.ltd/robots.txt :

In Plesk editor :

With my web browser :

 

[Aleksander Prikhozhdenko]

Hello,
Toolkit generate virtual robots.txt if you are switching off search engine visibling. It will be following:

<meta name='robots' content='noindex,follow' />

I suppose that in your case you have Yoast seo plugin installed in the WordPress. For investigation I did following:
- Install a clean WordPress
- Check that robots.txt is absent
- Install Yoast seo and perform initial configuration
- Check robots.txt.
Robots.txt looks like yours:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

So looks like your robots.txt was generated by plugin.

If you want to change your own robots.txt you can create a file robots.txt. It should replace WordPress virtual robots.txt

The default setting is the following...

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

First of all, there is NO documentation on this file ANYWHERE.
Where is it?
How do you change it or disable it?
Why does it replace the robots.txt file I already have?
Any answers would much appreciated. Thank you.

 

[WhatWouldJesusDo]

It was a clean install...using wordpress toolkit.
The virtual robots.txt file was generated immediately after install.
I added my own and nothing changed...
I THEN installed yoast...and nothing changed.

 

[Aleksander Prikhozhdenko]

Can you reproduce it at another domain or subdomain?
- Install a new WordPress via Toolkit at a new subdomain
- Make sure that virtual robots.txt is generated for the new WordPress right after installation, without any plugins

It was a clean install...using wordpress toolkit.
The virtual robots.txt file was generated immediately after install.
I added my own and nothing changed...
I THEN installed yoast...and nothing changed.