Browser caching with nginx not working

[paWELL]

Hi there,

i'm using:
Operating System: ‪CentOS 6.7 (Final)‬
Plesk Version: 12.5.30 Update #24

and have the following nginx settings:

Smart static files processing [on]

Serve static files directly by nginx [on]

ac3 avi bmp bz2 cue dat doc docx dts exe flv gz htm html ico img iso mkv mp3 mp4 mpeg mpg ogg pdf png ppt pptx qt rar rm swf tar tgz txt wav xls xlsx zip

Additional nginx directives


if ($scheme !~* ^https ){
rewrite ^ https://www.$server_name$request_uri? permanent;
}

gzip on;
gzip_proxied any;
gzip_types text/plain text/xml text/css application/javascript application/x-javascript text/javascript image/svg+xml video/mp4;
gzip_vary on;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";

location ~* \.(js|css|png|jpg|jpeg|gif|ico|woff|svg)$ {
expires 30d;
add_header Pragma public;
add_header Cache-Control "public";
try_files $uri @fallback;
}

### WP Super Cache Below ###

set $cache_uri $request_uri;

# POST requests and urls with a query string should always go to PHP

if ($request_method = POST) {

set $cache_uri 'null cache';

}

if ($query_string != "") {

set $cache_uri 'null cache';

}

# Don't cache uris containing the following segments

if ($request_uri ~* "(/wp-admin/|/xmlrpc.php|/wp-(app|cron|login|register|mail).php|wp-.*.php|/feed/|index.php|wp-comments-popup.php|wp-links-opml.php|wp-locations.php|sitemap(_index)?.xml|[a-z0-9_-]+-sitemap([0-9]+)?.xml)") {

set $cache_uri 'null cache';

}

# Don't use the cache for logged in users or recent commenters

if ($http_cookie ~* "comment_author|wordpress_[a-f0-9]+|wp-postpass|wordpress_logged_in") {

set $cache_uri 'null cache';

}

# Use cached or actual file if they exists, otherwise pass request to WordPress

location ~ / {

try_files /wp-content/cache/supercache/$http_host/$cache_uri/index.html $uri $uri/ /index.php ;

}

# all other requests go to WordPress

if (!-e $request_filename) {

rewrite . /index.php last;

}

______________________

But it doesn't work:

In Google Page Speed, it says there is no browser caching enabled.

What's the problem?

Thanks, for your help.

 

[Lloyd_mcse]

Hi paWELL,
have you tried removing the file types you want to be cached from the "Serve static files By Nginx" block?

eg: remove css, png

I hope that helps
Regards

Lloyd

 

[paWELL]

Hi paWELL,
have you tried removing the file types you want to be cached from the "Serve static files By Nginx" block?

eg: remove css, png

I hope that helps
Regards

Lloyd

Yes, all the files are removed.

 

[bulent]

this problem is reported months ago, so far no progress.
https://talk.plesk.com/threads/nginx-browser-caching-for-static-resources.334497/

 

[paWELL]

Hope this will be fixed soon. No workaround. I'm using Nginx as a reverse proxy...

 

[danami]

+1 to this. I really don't want to have to create custom nginx templates just to add proper caching to nginx.

 

[trialotto]

@paWELL and @bulent,

I suppose that it is related to the additional Nginx directives that have been applied (and no, I will not ramble about the stanza used, with comments like "if is evil in Nginx").

I am pretty sure that the additional Nginx directives will not work, due to the basic method of redirecting traffic to Nginx (which is kind of specific in Plesk).

On the other hand, there should not be any reason why the expires directive does not work.

I would like to ask you both to provide

- a brief summary of settings and choices made (and objectives)
- additional information (if any)
- the location (read: name of the specific file) in which the additional Nginx directives were used
- every other relevant aspect

so I can start working on an analysis of the root cause of the problem and, hopefully, a solution.

Regards.....

 

[paWELL]

@paWELL and @bulent,

I would like to ask you both to provide

- a brief summary of settings and choices made (and objectives)
- additional information (if any)
- the location (read: name of the specific file) in which the additional Nginx directives were used
- every other relevant aspect

so I can start working on an analysis of the root cause of the problem and, hopefully, a solution.

Regards.....

The summary of settings are in my first post. All the Settings where made in the plesk panel. No files edited:

Domains > my domain > Apache & nginx settings

 

[G J Piper]

I've found that it seems there is currently only one way to get nginx proxy and expires to work together on static files — and it involves modifying Plesk UI controlled files in the command line.
(of course, as a result any changes to Plesk's nginx settings will overwrite these settings)

Go into settings in the Domain -> Apache & nginx Settings and make sure these are checked:
[on] Smart static files processing
[on] Serve static files directly by nginx

Then go into the server files and find this file if you have more than just the default domain configured:
/var/www/vhosts/system/domain.ext/conf/nginx.conf

or this file if you only use the default domain:
/var/www/vhosts/system/defaultdomain.ext/conf/nginx_ip_default.conf

In one of these files (whichever one applies to you) look for the two sections that look like this:
location ~ ^/(.*\.(ac3|avi|bmp|bz2|css|cue|dat|doc|docx|dts|eot|exe|flv|gif|gz|htm|html|ico|img|iso|jpeg|jpg|js|mkv|mp3|mp4|mpeg|mpg|ogg|pdf|png|ppt|pptx|qt|rar|rm|svg|swf|tar|tgz|ttf|txt|wav|woff|woff2|xls|xlsx|zip))$ {
try_files $uri @fallback;
}

Make them both look like this:
location ~ ^/(.*\.(ac3|avi|bmp|bz2|css|cue|dat|doc|docx|dts|eot|exe|flv|gif|gz|htm|html|ico|img|iso|jpeg|jpg|js|mkv|mp3|mp4|mpeg|mpg|ogg|pdf|png|ppt|pptx|qt|rar|rm|svg|swf|tar|tgz|ttf|txt|wav|woff|woff2|xls|xlsx|zip))$ {
expires 7d;
try_files $uri @fallback;
}

There are two of these sections in the file because one is for SSL vhost settings and one is for regular vhost settings.

Then restart nginx by:
service nginx restart

Voila! You have nginx reverse-proxy and expires on any of the static files listed for the domain you modified.

Keep in mind -- if you change any settings in your "Apache & nginx Settings" for that domain you will have to redo these customizations manually, again.
I know it isn't an easy way to do it, especially when you have a lot of domains, but it seems to be the only way to get everse-proxy static files served with expires tags included.

The reason these settings have to be manually added is that once a "location" directive is made on a certain group of files, if those files are found in another "location" directive they are ignored by nginx.

Seems like it would be extremely simple for Plesk to include another setting next to the static files preferences in the UI that would allow enabling or disabling expires at least.

Hope this helps.

 

[trialotto]

@G J Piper,

I hope that you will not take any personal offense, but note the following:

a) the config files in /etc/nginx/plesk.conf.d/vhosts and /etc/nginx/plesk.conf.d/ip_default are symlinks to /var/www/vhosts/system/<sub>.<domain>.<tld>/conf/nginx.conf

b) the before mentioned nginx.conf file contains the include directive with

/var/www/vhosts/system/<sub>.<domain>.<tld>/conf/vhost_nginx.conf

c) the before mentioned vhost_nginx.conf file contains the contents of the additional nginx directives that can be entered via the Plesk Panel:

go to "Domains > [ domain ] > Apache & nginx Settings > (section) Additional Nginx directives"

and note that, if you press OK (in the Plesk Panel), the directives are checked and the (checked) custom configuration is reloaded automatically (i.e. nginx -t && service nginx reload).


In short, Nginx can be customized via the Plesk Panel.

Regards.......

 

[G J Piper]

@G J Piper,
a) the config files in /etc/nginx/plesk.conf.d/vhosts and /etc/nginx/plesk.conf.d/ip_default are symlinks to /var/www/vhosts/system/<sub>.<domain>.<tld>/conf/nginx.conf

Duly noted. I corrected my post.

b) the before mentioned nginx.conf file contains the include directive with
/var/www/vhosts/system/<sub>.<domain>.<tld>/conf/vhost_nginx.conf

c) the before mentioned vhost_nginx.conf file contains the contents of the additional nginx directives that can be entered via the Plesk Panel:
go to "Domains > [ domain ] > Apache & nginx Settings > (section) Additional Nginx directives"
and note that, if you press OK (in the Plesk Panel), the directives are checked and the (checked) custom configuration is reloaded automatically (i.e. nginx -t && service nginx reload).
In short, Nginx can be customized via the Plesk Panel.

I tried that at first too, but it doesn't work. The reason is this:
That include directive is found after the other "location" directive's brackets, which is why additional "location" directives for the static files added in the Plesk UI don't work. The expires setting must go inside the original location directive or it can't be applied to the same files. Nginx ignores any location directives for the same files after the first one.


Addition:
I created a two liner code to remove and add the desired settings to the correct files on my server:

To remove the expires settings, or to make sure no expires settings exist before adding them to all domains:
find /var/www/vhosts/system/*/conf/ -name 'nginx.conf' -exec sed -i '/expires /d' {} \;

To add the expires settings to all domains on the server:
find /var/www/vhosts/system/*/conf/ -name 'nginx.conf' -exec sed -i 's/try_files \$uri \@fallback;/expires 7d;\n\t\ttry_files \$uri @fallback;/g' {} \;

Let me know if anyone sees any problems with these and I'll modify them or remove them.

 

[trialotto]

@G J Piper,

Nginx is not that picky about the order of location directives, it just has a look whether matches are present.

Sure, Nginx seems to be the odd-ball, but it actually is not: the "matching" is done in a very intelligent and efficient fashion.

I will post a more general solution and explanation, so everyone can have a proper solution, that works with Plesk.

Regards........

 

[G J Piper]

@G J Piper,
Nginx is not that picky about the order of location directives, it just has a look whether matches are present.

If the order does not matter to nginx, but matches are present, how does it pick which one to ignore? (or are you saying it ignores both?)

 

[trialotto]

@paWELL (and @everyone),

In order to answer your question

What's the problem?

I must address a number of separate topics, which are essentially general guidelines, comments and solutions.

For that reason, I will create a number of sections, to deal with each topic individually.

A - WP Super Cache

The Nginx configuration for the WP Super Cache plugin is, in essence:

- unnecessary (read: not needed, I will explain below)
- bad (read: not good in many dimensions and I promised not to moan about "if is evil")
- often contradicts other directives (read: other directives will not work properly)

and so on.

Note that the $cache_uri of WP Super Cache is a not native to Nginx, it is simply a variable "made up" by WP Super Cache, which variable is

- equal to the (Nginx native) $request_uri variable, (and)
- will be set to a (different) text value (often "null cache") for specific patterns that match Nginx blocks (read: mostly location blocks).

However, if you would never create and/or set the variable $cache_uri in the first place, there is no need to set the $cache_uri variable to a (different) text value (like "null cache").

Also note that the Plesk default configuration is rather "smart", in the sense that Nginx will pass the $host variable as the host name of the "host request header" to Apache.

Simply stated, Apache does all the work, no need to mess with "made up" variables, like the $cache_uri variable.

In short, the whole part of Nginx configuration concerning WP Super Cache can be removed and even should be removed.

Note that you can also see this differently: if Nginx does not do any caching, no need to apply Nginx directives to turn off caching for specific matches (such as php scripts).

The above does imply that, if you have some kind of caching enabled on Nginx (which is not the case in Plesk), then you should add specific Nginx directives (this is off-topic).

B - Forcing https

One should not use "rewrites" or "if", in order to enforce a redirect to HTTPS.

Instead, use

server {
listen ...:80;
server_name ....;
server_name ....;

return 301 https://<domain>.<tld>$request_uri;

....
}

and the test in italics is what has to be added to the nginx.conf file of the specific domain (the remainder of the text is given to indicate where the line has to be inserted).

A problem is that the line cannot be added to the vhost_nginx.conf file, with the obvious disadvantage that it can occur that you have to re-insert the line again, if nginx.conf is updated.

C - Gzip compression

In general, one has to admit that gzip compression seems to be a good thing to do, but actually, it can be the case that it is not.

You should always verify that your TTFB (a measure of "wait time") is not becoming too high, when applying gzip.

Moreover, if your site consists of many smaller files, then it often is not worthwhile to enable gzip compression (since the files are small enough).

The above simply implies that one better searches for a way to reduce the size of files, before searching a method to compress files.

Nevertheless, a normal set of gzip directives would be:

gzip on;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";
gzip_proxied any;
gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon image/bmp image/svg+xml;
gzip_vary on;
gzip_comp_level 6;
gzip_buffers 16 8k;

and note the directive gzip_comp_level, which can be used to reduce (i.e. a lower value) or increase (a higher value) the compression level (range: 1 to 9).

This ability to set the compression level is quite effective, when battling long TTFBs.

Note that you can set the gzip related directives per domain (i.e. use the Plesk Panel, to indirectly edit vhost_nginx.conf) or on a server-wide level (edit /etc/nginx/conf.d/nginx.conf).

D - Allow for (browser) caching

Well, in general, there are many rules that we have to take into consideration.

I decided to keep it simple and provide a sufficient example to set some (relevant) headers for specific files:

# cache.appcache, your document html and data
# you should never cache this data, so apply a negative value, which is equel to stating "non-cache"
location ~ \.(manifest|appcache|html?|xml|json)$ {
expires -1;
access_log /var/www/vhosts/system/test.timeforwood.nu/logs/static.log;
}

# Feed
location ~ \.(rss|atom)$ {
expires 1h;
add_header Cache-Control "public";
}

# Media: images, icons
location ~* \.(jpeg|jpg|gif|png|ico|cur|gz|svg|svgz|webp)$ {
expires 1M;
access_log off;
add_header Cache-Control "public";
}

# Media: video, audio, HTC
location ~* \.(mp3|mpeg|mpg|mp4|ogg|ogv|webm|webp|htc)$ {
expires 3M;
access_log off;
add_header Cache-Control "public";
}

# CSS and Javascript
location ~* \.(css|js)$ {
expires 1y;
access_log off;
add_header Cache-Control "public";
# add_header Access-Control-Allow-Origin "*";
}

# CORS configuration for fonts
# this is useful when you use webfonts that are hosted elsewhere.
# the drawback is that there is a (minor) security penalty
#location ~ \.(ttf|ttc|otf|eot|woff|woff2|font.css)$ {
# add_header Access-Control-Allow-Origin "*";
#}


E - Conclusion

My conclusion is that my hands are feeling numb from the typing.

However, feel free to ask questions and I certainly hope that the above has helped a little bit.

Regards!

 

[trialotto]

If the order does not matter to nginx, but matches are present, how does it pick which one to ignore? (or are you saying it ignores both?)

@G J Piper

Nginx actually does a "best pattern match", it chooses the pattern that matches best and takes action. This is a somewhat simplified version of what actually happens.

One golden rule that not has to be forgotten is the following: if no match is found, the block "location / ... { }" is being used.

In short, it reads all "possible matches" and compares those to the URI requested.

I have to add that there is a substantial difference between "possible matches" and "patterns" (as in, for instance, patterns in the location blocks).

Consider the return directive: if that matches, Nginx simply executes the redirect and leaves.

Also consider the following location block: location \.php { }, where php is the pattern, which does not have to be a "possible match". A jpeg file does not match, so Nginx leaves.

All in all, it is just a sort of "if yes, then proceed. If no, look elsewhere". But in a very ingenious way: Nginx has some superior process logic.

Hope the above explains a bit (my apologies, it is 06:00 in the night/morning over here, I am getting somewhat tired now. Not so sharp anymore).

Regards...........

 

[Lloyd_mcse]

Addition:
To add the expires settings to all domains on the server:
find /var/www/vhosts/system/*/conf/ -name 'nginx.conf' -exec sed -i 's/try_files \$uri \@fallback;/expires 7d;\n\t\ttry_files \$uri @fallback;/g' {} \;

Why not just create custom templates then the conf wil be preserved...

/opt/psa/admin/conf/templates/custom/domain/nginxDomainVirtualHost.php

Line 105...

    location ~ ^/(.*\.(<?php echo $VAR->domain->physicalHosting->proxySettings['nginxStaticExtensions'] ?>))$ {
             try_files $uri @fallback;
}

And change it too (example only)...

    location ~ ^/(.*\.(<?php echo $VAR->domain->physicalHosting->proxySettings['nginxStaticExtensions'] ?>))$ {
        expires 7d;
        add_header Pragma public;
        add_header Cache-Control "public";
        try_files $uri @fallback;
    }

Then add gzip on a domain level.
Just a thought.
Regards

 

[danami]

Why not just create custom templates then the conf wil be preserved...
/opt/psa/admin/conf/templates/custom/domain/nginxDomainVirtualHost.php

I think that's the whole point of all the threads on this issue. It looks like if you want to set expires globally on nginx you currently need to either create custom nginx virtualhost templates or add the expires rules to every single vhosts nginx.conf.

Both options are a real pain when you have a lot of servers with a lot of domains. I'm not a fan of custom templates as the Plesk updater will complain when it detects them when updating and you need to always monitor diffs and check if Plesk made any changes in the future.

In its current form I don't see any other options without Plesk making the changes themselves to allow for this sort of thing. Maybe some sort of global include for nginx gets included in everyones virtualhost that fixes this sort of thing?

Also note that gzip can be applied globally with no problems See:
https://kb.plesk.com/en/122628

It's only the expires that has issues as it needs to be within the location block.

 

[G J Piper]

I'm not inclined to risk an update fiasco by making a custom template either. I've seen too many problems others have had with that.

I'm not super interested in the gzip option either as it creates overhead on every page served, and I optimize my static sites pretty well anyway.
I use WP Super Cache to save pre-gzipped static versions of pages on all WordPress domains as well so the nginx gzip option would be a step slower for that.

The expires settings are enough for me.

Simply having a text-area added to the form to insert settings into the location block that is created by the "Apache & nginx Settings" UI would be awesome — seems like it would be a really simple thing for Plesk to implement as well.

 

[bulent]

For me the easiest solution for the problem was to begin using Cloudflare and control the expire settings from their website.
Also it has some good stuff such as minify css, js and html.

 

[trialotto]

Why not just create custom templates then the conf wil be preserved...

/opt/psa/admin/conf/templates/custom/domain/nginxDomainVirtualHost.php

Line 105...

    location ~ ^/(.*\.(<?php echo $VAR->domain->physicalHosting->proxySettings['nginxStaticExtensions'] ?>))$ {
             try_files $uri @fallback;
}

And change it too...

    location ~ ^/(.*\.(<?php echo $VAR->domain->physicalHosting->proxySettings['nginxStaticExtensions'] ?>))$ {
        expires 7d;
        add_header Pragma public;
        add_header Cache-Control "public";
        try_files $uri @fallback;
    }

Then add gzip on a domain level.
Just a thought.
Regards

@Lloyd_mcse

This is a "probable concept", but a "bad idea": custom templates can be created, but even when disregarding some general disadvantages, the main problem with custom templates is that they are generating the main Nginx configuration file for domains, which implies that custom templates can overrule other custom settings, for instance those via vhost_nginx.conf.

Also note that you cannot permit mistakes in a custom template: it is not checked automatically AND Nginx can break completely with a badly configured custom template.

Furthermore, note that your specific example is rather restrictive: it does not allow for multiple (browser) caching settings, they are all set to 7d, no distinction is made.

In short, one has to be very careful, when using custom templates for Nginx.

Regards....