Resolved rel="canonical" response header is lost between Apache and Nginx proxy

[Deleted member 209767]

Server operating system version

AlmaLinux

Plesk version and microupdate number

latest

We have this problem on 5 servers running mod_pagespeed. The images are being converted to webp, but mod_pagespeed is rewriting the file name, and it changes continuously. This means that google indexes our webp images, and those images are lost after a few days when the file changes to a different name. This is negatively impacting our SEO.

To prevent this problem, mod_pagespeed is supposed to add a rel=canonical header to the requests. But somehow this header is lost between Apache and Nginx

When making a curl request directly to Apache without nginx proxy, I can see the header:

# curl -lv http://www.mysite.com:7080/someimage.png.pagespeed.ic.zDk0iIUL8D.webp
* Trying 167.114.64.16...
* TCP_NODELAY set
* Connected to www.mysite.com (167.114.64.16) port 7080 (#0)
> GET /someimage.png.pagespeed.ic.zDk0iIUL8D.webp HTTP/1.1
> Host: www.mysite.com:7080
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 21 Mar 2023 18:21:51 GMT
< Server: Apache
< Last-Modified: Wed, 15 Jun 2022 04:20:48 GMT
< Accept-Ranges: bytes
< Content-Length: 3438
< Link: <http://www.mysite.com:7080/someimage.png>; rel="canonical"
< X-Content-Type-Options: nosniff
< Expires: Tue, 21 Mar 2023 18:26:51 GMT
< Cache-Control: max-age=300,private
< Vary: Accept-Encoding
< Connection: close
< Content-Type: image/png

But when doing the same request to nginx proxy, the "Link" header is lost, same when checking in the dev console. Of course I tried to disable Cloudflare and all caching stuff, reset htaccess, remove anything modifying the headers, etc...

In Plesk, Smart static files processing is disabled, as recommended when running mod_pagespeed



We have been trying to fix this issue for months, and would appreciate some help! Thanks

 

[Deleted member 209767]

Any help?

 

[Deleted member 209767]

I see the canonical link header is not the only stripped one.
The Accept-Ranges: bytes, Pragma: public and Referrer-Policy: no-referrer-when-downgrade get stripped too in the curl request

 

[abashurov]

I have tried setting a test instance in a similar fashion, however, it returned all the expected headers, as nginx does not hide them by default: https://github.com/nginx/nginx/blob...src/http/modules/ngx_http_proxy_module.c#L836.

Do you have any custom nginx directives by any chance, any extensions adding configuration to /var/www/vhosts/example.com/conf/nginx.conf, or maybe some relevant output in nginx -T | grep proxy_hide_header?

 

[Deleted member 209767]

Hello, thanks for the reply

nginx -T | grep proxy_hide_header

[root@ns508000 nginx]# nginx -T | grep proxy_hide_header
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;
proxy_hide_header upgrade;

We have a bunch of nginx directives in Plesk, but mostly related to mod_pagespeed

<Location /pagespeed_admin>
Order allow,deny
Allow from all
SetHandler pagespeed_admin
</Location>
<Location /mod_pagespeed_statistics>
Order allow,deny
Allow from all
SetHandler mod_pagespeed_statistics
</Location>
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 100
ModPagespeed On
ModPagespeedEnableCachePurge on
ModPagespeedStatistics off
ModPagespeedStatisticsLogging off
ModPagespeedLogDir /var/log/pagespeed/ngnm
ModPagespeedSslCertDirectory "/etc/pki/tls/certs"
ModPagespeedSslCertFile "/etc/pki/tls/cert.pem"
ModPagespeedLoadFromFile "Overview" \
"/var/www/vhosts/site.com/public_html/"
ModPagespeedMapRewriteDomain "Overview" "http://localhost:7080"
ModPagespeedRateLimitBackgroundFetches off
ModPagespeedRewriteDeadlinePerFlushMs 10
ModPagespeedLRUCacheKbPerProcess 1024
ModPagespeedLRUCacheByteLimit 16384
ModPagespeedFileCachePath "/var/cache/mod_pagespeed/"
ModPagespeedFileCacheSizeKb 100000000
ModPagespeedFileCacheCleanIntervalMs 864000000
ModPagespeedFileCacheInodeLimit 50000000
ModPagespeedLoadFromFileCacheTtlMs 31556952000
ModPagespeedMemcachedServers localhost:11211
ModPagespeedEnableFilters rewrite_domains
ModPagespeedImagePreserveURLs off
ModPagespeedEnableFilters in_place_optimize_for_browser
ModPagespeedInPlaceResourceOptimization on
ModPagespeedSupportNoScriptEnabled false
ModPagespeedEnableFilters rewrite_css
ModPagespeedEnableFilters rewrite_style_attributes
ModPagespeedDisableFilters inline_css
ModPagespeedDisableFilters outline_css
ModPagespeedDisableFilters combine_css
ModPagespeedDisableFilters prioritize_critical_css
ModPagespeedEnableFilters recompress_images
ModPagespeedEnableFilters convert_png_to_jpeg
ModPagespeedEnableFilters convert_jpeg_to_progressive
ModPagespeedEnableFilters convert_jpeg_to_webp
ModPagespeedEnableFilters convert_to_webp_lossless
ModPagespeedEnableFilters recompress_webp
ModPagespeedDisableFilters lazyload_images
ModPagespeedEnableFilters resize_images
ModPagespeedEnableFilters resize_rendered_image_dimensions
ModPagespeedDisableFilters inline_images
ModPagespeedEnableFilters strip_image_meta_data
ModPagespeedEnableFilters insert_image_dimensions
ModPagespeedDisableFilters inline_preview_images
ModPagespeedImageRecompressionQuality 85
ModPagespeedJpegRecompressionQuality 85
ModPagespeedJpegRecompressionQualityForSmallScreens 70
ModPagespeedWebpRecompressionQuality 80
ModPagespeedWebpRecompressionQualityForSmallScreens 70
ModPagespeedEnableFilters collapse_whitespace
ModPagespeedEnableFilters remove_comments
ModPagespeedLowercaseHtmlNames on
ModPagespeedEnableFilters rewrite_javascript
ModPagespeedDisableFilters defer_javascript
ModPagespeedDisableFilters outline_javascript
ModPagespeedDisableFilters combine_javascript
ModPagespeedDisallow "*/ajax_shop.js"
ModPagespeedDisallow "*/ajax_basketid.js"
ModPagespeedDisallow "*/ajax_get_stock_state.js"
ModPagespeedDisallow "*/basket_size.js"
ModPagespeedDisallow "*/basket_size_ajax.js"
ModPagespeedDisallow "*/basket_api_get_basketitems.js"
ModPagespeedDisallow "*/basket_api.js"
ModPagespeedDisallow "*/basket_product.js"
ModPagespeedDisallow "*/basket_hover.js"
ModPagespeedEnableFilters canonicalize_javascript_libraries
ModPagespeedEnableFilters extend_cache
ModPagespeedModifyCachingHeaders on
ModPagespeedEnableFilters hint_preload_subresources
ModPagespeedPreserveSubresourceHints on
AddOutputFilterByType DEFLATE text/plain text/html text/xml;
AddOutputFilterByType DEFLATE text/css text/javascript;
AddOutputFilterByType DEFLATE application/xml application/xhtml+xml;
AddOutputFilterByType DEFLATE application/rss+xml;
AddOutputFilterByType DEFLATE application/javascript application/x-javascript
ModPagespeedForbidFilters combine_css,combine_javascript
ModPagespeedUrlValuedAttribute a data-thumb image

SetEnv no-gzip 1
ProxyErrorOverride on
ModPagespeedMaxSegmentLength 250
ModPagespeedEnableFilters insert_image_dimensions

proxy_connect_timeout 900;
proxy_send_timeout 900;
fastcgi_send_timeout 900;
fastcgi_read_timeout 900;

brotli on;
brotli_types text/xml
image/svg+xml
application/x-font-ttf
image/vnd.microsoft.icon
application/x-font-opentype
application/json
font/eot
application/vnd.ms-fontobject
application/javascript
font/otf
application/xml
application/xhtml+xml
text/javascript
application/x-javascript
text/plain
application/x-font-truetype
application/xml+rss
image/x-icon
font/opentype
text/css
image/x-win-bitmap;
brotli_comp_level 4;

proxy_http_version 1.1;
proxy_socket_keepalive on;
proxy_buffering off;

I tried removing the directives but still can't see the headers


/etc/nginx/nginx.conf

#user nginx;
worker_processes 16;

#error_log /var/log/nginx/error.log;
#error_log /var/log/nginx/error.log notice;
#error_log /var/log/nginx/error.log info;

#pid /var/run/nginx.pid;

include /etc/nginx/modules.conf.d/*.conf;

events {
worker_connections 1024; #L
#worker_connections 512;#L
accept_mutex on; #L
use epoll; #L
multi_accept on;#L

}


http {


pagespeed off;

include mime.types;
default_type application/octet-stream;

#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';

#access_log /var/log/nginx/access.log main;
# Added by L to get the user ip in logs
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 131.0.72.0/22;
set_real_ip_from 104.16.0.0/13;
set_real_ip_from 104.24.0.0/14;
set_real_ip_from 2400:cb00::/32;
set_real_ip_from 2606:4700::/32;
set_real_ip_from 2803:f800::/32;
set_real_ip_from 2405:b500::/32;
set_real_ip_from 2405:8100::/32;
set_real_ip_from 2a06:98c0::/29;
set_real_ip_from 2c0f:f248::/32;
real_ip_header CF-Connecting-IP;
# Enabled or set by L
tcp_nopush on;
tcp_nodelay on;
aio_write on;
postpone_output 0;
keepalive_timeout 600s 600s; #Clouflare need largue amount of time
keepalive_requests 20000;
output_buffers 128 4k;
# Until here

sendfile on;

#keepalive_timeout 0;
#keepalive_timeout 65;

gzip on;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";

#brotli on;
#brotli_comp_level 4;
#brotli_types text/xml image/svg+xml application/x-font-ttf image/vnd.microsoft.icon application/x-font-opentype application/json font/eot application/vnd.ms-fontobject application/javascript f$

proxy_buffers 8 16k;
proxy_buffer_size 32k;
fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;

server_tokens off;

include /etc/nginx/conf.d/*.conf;
}

# override global parameters e.g. worker_rlimit_nofile
include /etc/nginx/*global_params;

 

[mow]

Does the request even reach apache or does the pagespeed module in nginx deliver this?

 

[Deleted member 209767]

I have finally found the source of the problem - it is the LoadFromFile directive in mod_pagespeed config which loads resources from files instead of doing an HTTP request

After removing these directives, I can finally see the canonical header.

I have not found a solution for this, I think the only workaround is to completely disable LoadFromFile, unfortunately, it come at a cost of less efficiency