Question How to remove the HTML body of a 301 return of NGINX?

blueberry · Sep 16, 2020

When I perform a "curl h t t p : / / talk.plesk.com"

I get:
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

This is the default behavior of nginx for a 301 redirect. The same happens on my websites and I would like to get rid of this HTML body to save bytes and also because it is barely in compliance with the rfc7231

RFC 7231 - Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content

he server SHOULD generate a Location header field in the response
containing a preferred URI reference for the new permanent URI. The
user agent MAY use the Location field value for automatic
redirection. The server's response payload usually contains a short
hypertext note with a hyperlink to the new URI(s).

As you can see in the default response of nginx there is no <a href="new location">here</a> link. If you do a curl triggering a 301 redirection from apache you will see the response contains a A HREF with the destination URL in it.

I which I could only have the HTTP header fields: curl h t t p : / / talk.plesk.com -I

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Wed, 16 Sep 2020 08:23:13 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://talk.plesk.forum

FYI, if you do "curl http://twitter.com" "curl ht tp://facebook.Com", nothing is going to get returned because twitter and facebook got rid of the HTML body of 301 redirect. They rely on the HTTP header field "location" and the "HTTP/1.1 301 Moved Permanently" in the HTTP header. I think it is great because bots don't spend their time looking in the HTML body of the 301, they directly rely on the location HTTP header field.

so if someone has a solution to get rid of the HTML body generated by the following:

I would be willing to know it.

Bitpalast · Sep 19, 2020

According to https://www.ietf.org/rfc/rfc2616.txt it is correct and desired behavior that a document body is sent along with a 301. See section 10.3.2.

In order to change the Nginx default behavior, it will be necessary to modify several things.

a) You need your own custom 301 .html page in your Nginx installation. That page should not have any content.

b) The server sections in the /etc/nginx/plesk.conf.d/server.conf file need an update so that the "Content-Type" line from the response is removed. However, that file is auto-created by Plesk, so changes can and will be overwritten on configuration changes by Plesk. You need to place the change into the configuration file template. That is something, I'd never ever do, because on the next update, it can cause issues with generating web server configuration files.

Code:

server {
... [other stuff by Plesk] ...
    error_page 301 302 @30x; 
    location @30x {
        default_type ""; # removes the Content-Type 
        return 300; # is needed but will never be used
    }
    return 301 http://<error page address of the empty error page>;
}

So by all means, personally I'd rather discourage anybody from attempting such modifications. They may be suitable on a server that you can freely configure, but on a panel system like Plesk it is probably not such a great idea to do.

blueberry · Sep 22, 2020

Thank you. I chose not to modify server config files. In the past, I already crashed my nginx configuration and hopefully I had config file backups. I will rely on Apache redirections for the pages and maybe no redirections at all for the static files (I have sent a question to google about this matter).

For those who are interested about the issue. Here is what I have observed.

I have a website and Some years ago I migrated from HTTP to HTTPS. At some point, I have decided to move the automated redirection from Apache to Nginx.
Something strange happened after several weeks. The traffic coming from Google plummeted by approximately 50%.
I reactivated the Apache redirection and the Google traffic went up again. I reactivated the Nginx redirection and it plummeted again.
So, I think that apache redirects are better than nginx redirects. Maybe, Googlebot does not solely rely on the location HTTP header field. Maybe googlebot is "bothered" by the content of the 301 HTML body.

HTML body of 301 Apache redirect:

Code:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://xxxxx.com/">here</a>.</p>
<hr>
<address>Apache Server at defenseofmen.com Port 80</address>
</body></html>

HTML body of 301 nginx redirect:

Code:

<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

I understood lately that the option " Permanent SEO-safe 301 redirect from HTTP to HTTPS" triggered 301 redirects from HTTP to HTTPs powered by NGINX.
That's why I faced this issue. That's why I prefer to untick this option. Since in my hosting settings apache is activate, The 301 redirect of my HTaccess file is taken into account for redirect from HTTP to HTTPs.

Nginx is still the only server used for delivering static files. If I activate the nginx redirection my CSS, JS, JPG files will be redirected using the nginx redirection. I don't know if it is a problem for Googlebot. So, that's why I asked John Mueller a question. It is a problem if our CSS, JS, JPG files are available from both a HTTP and HTTPS URL?
On plesk, I have a website powered by Nginx only and I have setup no nginx redirections at all. The canonical of the domain is in HTTPS. Quite interestingly, Doing like this and I had a lot of traffic for this website. CSS, JS, JPG files are available from HTTP and HTTPS for this website.

Bitpalast · Sep 22, 2020

Google does not at all include redirected URLs in its index.

blueberry · Sep 22, 2020

Here is the answer from Google about leaving static files accessible from HTTP and HTTPs on Plesk to prevent a NGINX 301 redirect in Additional nginx directives:

blueberry · Sep 24, 2020

So, in order to secure the static files of nginx, i have decided to remove access from all static files in HTTP mode by adding the following directives to the additional nginx directives:

Code:

location ~* .(js|jpg|jpeg|gif|png|css|tgz|gz|rar|bz2|doc|pdf|ppt|tar|wav|bmp|rtf|swf|ico|flv|txt|woff|woff2|svg)$ {
    expires 31d;
    add_header Pragma "public";
    add_header Cache-Control "public";

    if ($scheme = http) {
        return 403;
    }

}

After adding some headers, the directive checks if the request is in http and it returns a 403 in case it is a http rather than https. After thinking about it, there is no reason why somebody would link to the HTTP version of a URL to reach a CSS,JS,WOFF,JPG resource.
Meanwhile Apache 301 redirects the HTTP to HTTPS requests for all HTML pages (this is a wordpress installation)

mohammad umer · Sep 25, 2020

e have noticed that when using nginx internal 301 and 302 handling, nginx will serve a small document body with the appropriate Location: ... header.

Something along the lines of (in html): 301 redirect - nginx.

As appropriate in the above behaviour, a content-type text/html and content-length header is also sent.

We do a lot of 302 and some 301 redirects, the above behaviour is wasted bandwidth in our opinion.

Any way to disable this behaviour?

One idea that crossed our mind was to set error_page 301 302 to an empty text file. We have not tested this yet, but I am assuming even with the above, the content-type and content-length (0) headers will be sent.

So, is there a clean way to send a "body-less" 301/302 redirect with nginx?

blueberry · Sep 29, 2020

mohammad umer said:
e have noticed that when using nginx internal 301 and 302 handling, nginx will serve a small document body with the appropriate Location: ... header.

Something along the lines of (in html): 301 redirect - nginx.

As appropriate in the above behaviour, a content-type text/html and content-length header is also sent.

We do a lot of 302 and some 301 redirects, the above behaviour is wasted bandwidth in our opinion.

Any way to disable this behaviour?

One idea that crossed our mind was to set error_page 301 302 to an empty text file. We have not tested this yet, but I am assuming even with the above, the content-type and content-length (0) headers will be sent.

So, is there a clean way to send a "body-less" 301/302 redirect with nginx?

I found the way to make it work. The previous code from peter which is also the solution that published on serverfault https://serverfault.com/questions/421976/nginx-301-and-302-serving-small-nginx-document-body-any-way-to-remove-this-beha didn't work in server.conf,

it was not a bad thing to know that it didn't work because as he mentionned it is a bad idea to modify the server configuration files as plesk may generate new files and erase the previous one.

So, I have found a way to do it "the plesk way". I use Apache and Nginx as a proxy. So, my static files like my images are returned by Nginx instead of Apache.
Consequently, i have added a code in the Additional nginx directives

Code:

error_page 301 302 @30x;
location @30x {
    default_type ""; # removes the Content-Type
    return 300; # is needed but will never be used
}

location ~* .(js|css|tgz|gz|rar|bz2|doc|pdf|ppt|tar|wav|bmp|rtf|swf|ico|flv|woff|woff2|svg|jpg|jpeg|gif|png|txt)$ {
    expires 31d;
    add_header Pragma "public";
    add_header Cache-Control "public";

    if ($scheme = http) {
        return 301 https://$host$request_uri;
    }

}

Then, it will not return a hidden document in the 301 redirect when you perform a "curl http : // yourdomain.com / license . txt" if you have a wordpress website. It will only return the HTTP header fields with the location of the new URL only.

blueberry · Sep 29, 2020

I guess that it should work if you only use nginx. MAybe something like the following in additional nginx settings?
Does it work?

error_page 301 302 @30x;
location @30x {
default_type ""; # removes the Content-Type
return 300; # is needed but will never be used
}

if ($ssl_protocol = "") {
rewrite ^ https://$server_name$request_uri? permanent;
}

blueberry · Sep 29, 2020

By the way, I suspect Googlebot to have a bad behaviour when it reads this:
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

I have noticed that many http urls ended in my crawl anomaly category or soft 404.
So, I think that when googlebot finds a HTML body in a 301 redirect, he thinks that it is a real document with the wrong HTTP code (301 instead of 200)

I forgot to mention that the permanent SEO sage 301 redirection should remain unchecked otherwise it will take over the redirection:

Question How to remove the HTML body of a 301 return of NGINX?

blueberry

Basic Pleskian

Bitpalast

Plesk addicted!

blueberry

Basic Pleskian

Bitpalast

Plesk addicted!

blueberry

Basic Pleskian

blueberry

Basic Pleskian

mohammad umer

New Pleskian

blueberry

Basic Pleskian

blueberry

Basic Pleskian

Attachments

blueberry

Basic Pleskian

Similar threads