Wrong directory being indexed

[shane8020]

Hi, I hope someone can help because I've exhausted my little knowledge!

My site works perfectly but Google cant index pages. In the error_log I am getting loads of errors as below:

[error] [client 66.249.72.232] File does not exist: /var/www/vhosts/default/htdocs/data-storage
[error] [client 66.249.72.232] File does not exist: /var/www/vhosts/default/htdocs/networks
[error] [client 180.76.5.65] File does not exist: /var/www/vhosts/default/htdocs/components
[error] [client 66.249.72.232] File does not exist: /var/www/vhosts/default/htdocs/projectors
[error] [client 66.249.72.232] File does not exist: /var/www/vhosts/default/htdocs/projectors

My site isnt installed in /var/www/vhosts/default/htdocs it is installed in /var/www/vhosts/mysite.com/httpdocs

How can I change where to direct google etc to look at the correct site?

OS CentOS 5.7 (Final)
Panel version 11.0.9 Update #13


Thanks

 

[shane8020]

Can anyone help with this?

 

[IgorG]

Are you sure that your website is correctly visible from Internet?

 

[shane8020]

Hi, yes it is, I can view it and browse it and log into the admin, customers are also able to use it.

Hosting Settings for Document root is set to httpdocs in plesk

What else can I check

Thanks

 

[IgorG]

But when you try open this site somewhere - will it be recorded to domain's access_log?

 

[shane8020]

I.m not entirely sure, I visited the site and looked for the pages I visited and they dont exist, however the lines below do?


66.249.72.232 - - [13/Sep/2012:10:30:43 +0100] "GET /peripherals/mice/explorer-touch-mouse.html HTTP/1.1" 404 310 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.72.232 - - [13/Sep/2012:10:33:06 +0100] "GET /f1a75-m-le/reviews HTTP/1.1" 404 286 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.72.232 - - [13/Sep/2012:10:33:07 +0100] "GET /cameras/digital-cameras/finepix-l55-2.html HTTP/1.1" 404 310 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

 

[IgorG]

According to this log your site is under Google bot scanning.

 

[shane8020]

There is something wrong here. I assumed the 404 in the logs was an error?

Why is the vhosts/default/htdocs/ being scanned for links/pages that dont exist i.e projectors, data-storage, networks, components

Why cant Google access my site - Googlebot can't access your site

"Over the last 24 hours, Googlebot encountered 6793 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%."

As far as I understand without a robots.txt everything is indexed, with the file below everything is also indexed, I have tried both.

robots.tx content

User-agent: *
Allow: /

I think this all started when I upgraded to Plesk 11

 

[shane8020]

Can anyone help me out with this? Still getting loads of error and Google still cant crawl site?

 

[shane8020]

Could Support please reply to this, all my pages and products on google shopping cant be crawled with this problem, some support would be nice!

 

[IgorG]

Have you tried contact Support Team http://www.parallels.com/support/home/ ?

 

[DineshK]

Same problem

After I have moved to Plesk 11 Google is not indexing my pages. Can someone please help ASAP!!

 

[IgorG]

Have you tried disable nginx and check this issue with disabled nginx?