Plesk 12.5.30: nginx gives 403 forbidden on images after upgrade from 11.x

[rikay]

I'm running a Wordpress website and recently upgraded my VPS from 11.x to 12.5. Since the upgrade I get 403 forbidden from nginx after uploading images in Wordpress. I had no problem whatsoever with my setup on 11.x.

The weirdest thing is, all additional image sizes generated by Wordpress are displayed correctly, only the original image gives a 403. So screenshot.png will return a 403, but screenshot-150x150.png that is generated as a thumbnail by Wordpress won't. Also, I can't deleted or open the original images via FTP, gives permission denied as well.

My VPS
‪CentOS 6.7 (Final)‬
PHP support 5.4.45, run PHP as FPM application served by nginx
Smart static file processing on
All permissions are set to 755 for folders and 644 for files
User/owner: (changed my user to example-user)
All other server settings are pretty default (ootb)

 d root     root    /
d root     root    var
d root     root    www
d root     root    vhosts
d example-user psaserv domainname
d example-user psaserv httpdocs
d example-user psacln  wp-content
d example-user psacln  uploads
d example-user psacln  2015
d example-user psacln  11
- example-user psacln  screenshot-2.png

These settings worked out of the box on any Wordpress install on 11.x, but now even on a clean Wordpress install I get those 403 errors after an image upload. I have checked the default config files generated by Plesk but they seem to look okay. I've tried regenarting the config files but that didn't change anything. When disabling smart static file processing all images are served via Apache without 403 or error. Other static files like js and css files are served without error.

Are their any changes made in Plesk 12.5 regarding default config files or the way nginx handles images on upload through Wordpress?

 

[IgorG]

I would suggest you check vhost permissions with

# plesk repair fs

More detail can be found here - http://docs.plesk.com/en-US/12.5/ad...ility/plesk-repair-utility-file-system.74668/

 

[rikay]

All permissions are OK. I've tried it again on a clean install but to no succes.

Here is my error log when trying to upload a new image.

2015-12-24 13:38:45    Access    myip    200    GET /wp-admin/media-new.php HTTP/1.1    http://mydomain.nl/wp-admin/upload.php    Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36    25.6 K nginx access
2015-12-24 13:39:03    Access    myip    200    POST /wp-admin/admin-ajax.php HTTP/1.1    http://mydomain.nl/wp-admin/media-new.php    Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36    58 nginx access
2015-12-24 13:39:12    Access    myip    200    POST /wp-admin/async-upload.php HTTP/1.1    http://mydomain.nl/wp-admin/media-new.php    Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36    12 nginx access
2015-12-24 13:39:12    Access    myip    200    POST /wp-admin/async-upload.php HTTP/1.1    http://mydomain.nl/wp-admin/media-new.php    Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36    316 nginx access
2015-12-24 13:39:12    Access    myip    200    GET /wp-content/uploads/2015/12/logo-150x150.png HTTP/1.1    http://mydomain.nl/wp-admin/media-new.php    Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36    17.1 K nginx access
2015-12-24 13:39:15    Access    myip    200    GET /wp-admin/post.php?post=53&action=edit HTTP/1.1    http://mydomain.nl/wp-admin/media-new.php    Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36    43.4 K nginx access
2015-12-24 13:39:15    Error     myip    403    GET /wp-content/uploads/2015/12/logo.png HTTP/1.1    http://mydomain.nl/wp-admin/post.php?post=53&action=edit    Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36    564 nginx access
2015-12-24 13:39:15    Error     myip            7346#0: *191 open() "/var/www/vhosts/mydomain.nl/httpdocs/wp-content/uploads/2015/12/logo.png" failed (13: Permission denied) nginx error

Edit: Please notice how the 150x150 format of the original file is served without errors.

 

[Polsys]

Hi rikay. Absolutely identical problem here. I've spent several hours searching for a solution with zero success. Have you made any progress in your situation? I'll appreciate any help about this. Thanks!

I'm in Plesk 12.5.30 Update #44 & ‪CentOS 6.8 (Final)

 

[UFHH01]

Hi Polsys,

pls. consider to add your domain - specific error - log ( apache AND nginx, if you use both on your domain ) ( hint: it might be usefull to REDUCE a complete log and just post the complete ERROR - lines ), located at "/var/www/vhosts/system/DOMAIN-OR-SUBDOMAIN.COM/logs/", if you would like help with your investigations.

It is always a very good idea to post domain - specific configuration files from "/var/www/vhosts/system/DOMAIN-OR-SUBDOMAIN.COM/conf/" and possible ".htaccess" - files from your domain/subdomain - specific docroot, if you use apache without nginx.

 

[Richard18]

I had the same problem after WordPress multisite migration the other week.

Going to domain.com/site/files/image.png gave the 403 error.

Going to domain.com/wp-content/blogs.dir/blogid/files/image.png worked fine.

I guessed either something in the .htaccess or the wp-config.php were corrupt so I recreated them with the variables and directives from the suggested configs inside the WP dashboard.

Fixed it!

 

[Polsys]

I attach the error logs after today test.

After uploading test1.jpg with the WordPress built-in ajax uploader (same result with the standard uploader)

/var/www/vhosts/system/<DOMAIN>/logs/proxy_access_log says:

<CLIENT_IP> - - [24/Aug/2016:10:05:54 +0200] "POST /wp-admin/async-upload.php HTTP/1.1" 200 628 "http://www.<DOMAIN>/wp-admin/upload.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
<CLIENT_IP> - - [24/Aug/2016:10:05:54 +0200] "GET /wp-content/uploads/2016/08/test1-300x200.jpg HTTP/1.1" 200 11730 "http://www.<DOMAIN>/wp-admin/upload.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
<CLIENT_IP> - - [24/Aug/2016:10:05:56 +0200] "GET /wp-content/uploads/2016/08/test1-1024x682.jpg HTTP/1.1" 200 62050 "http://www.<DOMAIN>/wp-admin/upload.php?item=2660" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
<CLIENT_IP> - - [24/Aug/2016:10:06:25 +0200] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 58 "http://www.<DOMAIN>/wp-admin/upload.php?item=2660" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"
<CLIENT_IP> - - [24/Aug/2016:10:07:25 +0200] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 58 "http://www.<DOMAIN>/wp-admin/upload.php?item=2660" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"

After accessing test1.jpg whit a browser

/var/www/vhosts/system/<DOMAIN>/logs/proxy_error_log says:

2016/08/24 10:11:42 [error] 10887#0: *5894 open() "/var/www/vhosts/<DOMAIN>/httpdocs/wp-content/uploads/2016/08/test1.jpg" failed (13: Permission denied), client: <CLIENT_IP>, server: <DOMAIN>, request: "GET /wp-content/uploads/2016/08/test1.jpg HTTP/1.1", host: "www.<DOMAIN>"

Now with nginx OFF and the domain reconfigured to Apache without nginx

/var/www/vhosts/system/<DOMAIN>/logs/error_log says:

[Wed Aug 24 10:33:28 2016] [error] [client <CLIENT_IP>] (13)Permission denied: file permissions deny server access: /var/www/vhosts/<DOMAIN>/httpdocs/wp-content/uploads/2016/08/test1.jpg

This is the /var/www/vhosts/system/<DOMAIN>/conf/nginx.conf

#ATTENTION!
#
#DO NOT MODIFY THIS FILE BECAUSE IT WAS GENERATED AUTOMATICALLY,
#SO ALL YOUR CHANGES WILL BE LOST THE NEXT TIME THE FILE IS GENERATED.

server {
    listen <HOST_IP>:80;

    server_name <DOMAIN>;
    server_name www.<DOMAIN>;
    server_name ipv4.<DOMAIN>;

    client_max_body_size 128m;

    root "/var/www/vhosts/<DOMAIN>/httpdocs";
    access_log "/var/www/vhosts/system/<DOMAIN>/logs/proxy_access_log";
    error_log "/var/www/vhosts/system/<DOMAIN>/logs/proxy_error_log";

    location / {
        proxy_pass http://<HOST_IP>:7080;
        proxy_set_header Host             $host;
        proxy_set_header X-Real-IP        $remote_addr;
        proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header X-Accel-Internal /internal-nginx-static-location;
        access_log off;
    }

    location /internal-nginx-static-location/ {
        alias /var/www/vhosts/<DOMAIN>/httpdocs/;
        add_header X-Powered-By PleskLin;
        internal;
    }

    location ~ ^/(plesk-stat|webstat|webstat-ssl|ftpstat|anon_ftpstat|awstats-icon) {
        proxy_pass http://<HOST_IP>:7080;
        proxy_set_header Host             $host;
        proxy_set_header X-Real-IP        $remote_addr;
        proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header X-Accel-Internal /internal-nginx-static-location;
        access_log off;
    }

    location ~ ^/~(.+?)(/.*?\.php)(/.*)?$ {
        alias /var/www/vhosts/<DOMAIN>/web_users/$1/$2;
        fastcgi_split_path_info ^((?U).+\.php)(/?.+)$;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_pass "unix:///var/www/vhosts/system/<DOMAIN>/php-fpm.sock";
        include /etc/nginx/fastcgi.conf;
    }

    location ~ ^/~(.+?)(/.*)?$ {
        proxy_pass http://<HOST_IP>:7080;
        proxy_set_header Host             $host;
        proxy_set_header X-Real-IP        $remote_addr;
        proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header X-Accel-Internal /internal-nginx-static-location;
        access_log off;
    }

    location ~ \.php(/.*)?$ {
        fastcgi_split_path_info ^((?U).+\.php)(/?.+)$;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_pass "unix:///var/www/vhosts/system/<DOMAIN>/php-fpm.sock";
        include /etc/nginx/fastcgi.conf;
    }

    location ~ /$ {
        index index.html index.cgi index.pl index.php index.xhtml index.htm index.shtml;
    }

    include "/var/www/vhosts/system/<DOMAIN>/conf/vhost_nginx.conf";
}

This is the /var/www/vhosts/system/<DOMAIN>/conf/vhost_nginx.conf

location ~* \.(ac3|avi|bmp|bz2|css|cue|dat|doc|docx|dts|eot|exe|flv|gif|gz|htm|html|ico|img|iso|jpeg|jpg|js|mkv|mp3|mp4|mpeg|mpg|ogg|pdf|png|ppt|pptx|qt|rar|rm|svg|swf|tar|tgz|ttf|txt|wav|woff|woff2|xls|xlsx|zip)$ {
    expires 30d;
    add_header Pragma public;
    add_header Cache-Control "public";
    try_files $uri @fallback;
}

gzip on;
gzip_proxied any;
gzip_types text/plain text/xml text/css application/javascript application/x-javascript text/javascript image/svg+xml video/mp4;
gzip_vary on;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";

if (!-e $request_filename) {
    rewrite ^.*$ /index.php last;
}

Here is the users, groups and permissions of the file updated and the thumbnails auto-generated by WordPress:

-rw-r--r--. 1 sameusername psacln   62050 24 ago 10:05 test1-1024x682.jpg
-rw-r--r--. 1 sameusername psacln    5604 24 ago 10:05 test1-150x150.jpg
-rw-r--r--. 1 sameusername psacln   10655 24 ago 10:05 test1-270x200.jpg
-rw-r--r--. 1 sameusername psacln   11730 24 ago 10:05 test1-300x200.jpg
-rw-r--r--. 1 sameusername psacln   29846 24 ago 10:05 test1-600x400.jpg
-rw-r--r--. 1 sameusername psacln   31039 24 ago 10:05 test1-640x400.jpg
-rw-r--r--. 1 sameusername psacln   35505 24 ago 10:05 test1-675x450.jpg
-rw-r--r--. 1 sameusername psacln    2153 24 ago 10:05 test1-75x50.jpg
-rw-r--r--. 1 sameusername psacln   41654 24 ago 10:05 test1-768x512.jpg
-rw-r--r--. 1 sameusername psacln    2392 24 ago 10:05 test1-83x55.jpg
-rw-r--r--. 1 sameusername psacln  129580 24 ago 10:05 test1.jpg

The most disturbing fact for me is the 403 error only affects the original test1.jpg original file. The thumbnails (whit identical permissions and privileges) are working correctly.

I did a few more tests today and:

• Uploading files without nginx (pure apache) the problem does not appear.
  
• With other CMS (Prestashop) in the same server it seems the problem does not appear. Maybe the error only occurs with the mix of WordPress, Plesk & Nginx.

Please tell me if more information is needed.

 

[gennolo]

Same error here in the same identical configuration ...

I thought it had something to do with files header / mime type not being properly set by the Wordpress ajax upload,
but I did a lot of tests but I never found an explanation or a solution so I will follow the topic with particual interest.

 

[UFHH01]

Hi Polsys,
Hi gennolo,

consider to add the system-user "nginx" to the group "psacln", to solve your issues with permissions regarding apache + nginx (+PHP-FPM ).

For Ubuntu/Debian - based systems for example you would use the following command over your command line ( as root ):

usermod -aG psacln nginx​

 

[gennolo]

Just tried it... (I'm on CentOS but I think syntax is the same)

]# usermod -aG psacln nginx

I get :

usermod: user 'nginx' does not exist

But I currently see
"nginx : nginx psaserv"
in /etc/passwd
and a current process running as "nginx" user.
31436 nginx 30 10 53040 11m 2628 S 1.0 0.2 5:43.76 nginx

 

[UFHH01]

Hi gennolo,

I assume, that you use "copy&paste", instead of typing the command, provided here in the forum. Pls. be aware that "copy&paste" from a forum with formats may lead to issues on the command line ( invisible code/formats could be coppied as well ). Consider to TYPE the provided command, if you experience issues on your command line, when you perform "copy&paste", or copy the code/command from the forum to a text - editor and copy then afterwards the code/command from the text - editor, followed by the paste command on your command line with your ssh - client.

 

[gennolo]

Thank you - Yes - My mistake - I did copy-paste.
"nginx" was now added to psacln group... But still unsolved problem

Note that :

I attach the error logs after today test.

The most disturbing fact for me is the 403 error only affects the original test1.jpg original file. The thumbnails (whit identical permissions and privileges) are working correctly.

I did a few more tests today and:

• Uploading files without nginx (pure apache) the problem does not appear.
  
• With other CMS (Prestashop) in the same server it seems the problem does not appear. Maybe the error only occurs with the mix of WordPress, Plesk & Nginx.

IMHO this behaviour (the same happens to me) excludes it is a permission / group problem, Otherwise ALL images (that shares the same ownership/groupship) should return 403 Forbidden.
The problem only appears on images uploaded from the Wordpress Ajax uploader.
If the same image is loaded by FTP (to the same folder ! ) with the same permissions (user / owner) it is displayed correctly without 403...

 

[Polsys]

Some more tests and notes about this:

• To discard problems with a migrated websites, I have installed a fresh, blank WordPress instance into a new Plesk vhost. The error persists.
• With images uploaded with the browser non-ajax method, the error also appears. Is not an exclusive problem with the ajax uploader.
• I'm working in a new and fresh Plesk 12.5.30 cloud server, not in a Plesk upgraded from 11 or older.
• I made a test in other server with identical Plesk version and SO, in a different infrastructure, and this problem does not occur.

The user nginx is already in the psaserv group:

$ id nginx
uid=496(nginx) gid=497(nginx) grups=497(nginx),502(psaserv)

I will continue investigating...

 

[UFHH01]

Hi gennolo,

IMHO this behaviour (the same happens to me) excludes it is a permission / group problem

Wrong investigations, sorry and therefore the wrong conclusion. The error states CLEAR and without any possible interpretation: "(13)Permission denied: file permissions deny server access".


Pls. check your files/folder permissions under: "../wp-content/uploads" and as well the docroot and make sure, that they have the correct permissions: domain-system-user : psacln ( drwxr-xr-x = 0755 = folders // -rw-r--r-- = 0644 = files ).
The "domain-system-user" should be as well the very same user, who runs your domain-specific PHP-FPM - service

To correct files and folder permissions you could use:

For a domain:

chown -R domain-system-user:psacln /var/www/vhosts/YOUR-DOMAIN.COM/httpdocs
chown domain-system-user:psaserv /var/www/vhosts/YOUR-DOMAIN.COM/httpdocs

For a subdomain:

chown -R domain-system-user:psacln /var/www/vhosts/YOUR-DOMAIN.COM/SUB.YOUR-DOMAIN.COM
chown domain-system-user:psaserv /var/www/vhosts/YOUR-DOMAIN.COM/SUB.YOUR-DOMAIN.COM

What do these commands do?
The "chown" - command, with the additional command - string "-R" changes the given permissions for a user and group recursively for the given folders. Due to the fact that the docroot itself of your domain/subdomain has to be ownded by the group "psaserv", the second command is necessary, when you correct permissions for files/folders recursively as suggested.

 

[UFHH01]

The user nginx is already in the psaserv group:

Pls note:

To correct files and folder permissions you could use:

For a domain:

chown -R domain-system-user:psacln /var/www/vhosts/YOUR-DOMAIN.COM/httpdocs
chown domain-system-user:psaserv /var/www/vhosts/YOUR-DOMAIN.COM/httpdocs

For a subdomain:

chown -R domain-system-user:psacln /var/www/vhosts/YOUR-DOMAIN.COM/SUB.YOUR-DOMAIN.COM
chown domain-system-user:psaserv /var/www/vhosts/YOUR-DOMAIN.COM/SUB.YOUR-DOMAIN.COM

What do these commands do?
The "chown" - command, with the additional command - string "-R" changes the given permissions for a user and group recursively for the given folders. Due to the fact that the docroot itself of your domain/subdomain has to be ownded by the group "psaserv", the second command is necessary, when you correct permissions for files/folders recursively as suggested.

 

[gennolo]

Hi gennolo,

For a domain:

chown -R domain-system-user:psacln /var/www/vhosts/YOUR-DOMAIN.COM/httpdocs
chown domain-system-user:psaserv /var/www/vhosts/YOUR-DOMAIN.COM/httpdocs

For a subdomain:

chown -R domain-system-user:psacln /var/www/vhosts/YOUR-DOMAIN.COM/SUB.YOUR-DOMAIN.COM
chown domain-system-user:psaserv /var/www/vhosts/YOUR-DOMAIN.COM/SUB.YOUR-DOMAIN.COM

Thank you @UFHH01 for your prompt interest.

Permissions had been checked , double-checked , triple-checked .. I also launched "plesk repair fs" on the domain, just in case.
They had been also checked from shell. No reports of wrong permission.

All files are 644 with domain-owner and psacln group
All folders are 755 with domain-owner and psacln group
"httpdocs" is 755 with domain-owner and psaserv group

I would like to point out again that problem occur in the same folder (ie. /wp-content/uploads/2016/08/ )
For instance there are three different jpg files.
They all come with same owner / group (domain-user / psacln)

First file is returning 403 forbidden when requested from browser.
Other files are working OK. Permissions on files are identical.

The situation is exactly the same explained by Polysys in today's post.

The "403" file is the one loaded with Wordpress Ajax , the working files are either being uploaded by FTP manually
or generated by the Wordpress thumbnail generation system.

​

 

[Polsys]

A walk to a file with problems (httpdocs/wp-content/uploads/2016/08/test1.jpg)

drwxr-x---  <domain-system-user>  psaserv  httpdocs
drwxr-xr-x  <domain-system-user>  psacln   wp-content
drwxr-xr-x  <domain-system-user>  psacln   uploads
drwxr-xr-x  <domain-system-user>  psacln   2016
drwxr-xr-x  <domain-system-user>  psacln   08
-rw-r--r--  <domain-system-user>  psacln   test1.jpg

If now the user nginx is in...

$ id nginx
uid=496(nginx) gid=497(nginx) grups=497(nginx),502(psaserv)

...maybe I must to add the user nginx to the group psacln too?

 

[Polsys]

...maybe I must to add the user nginx to the group psacln too?

Sorry @UFHH01 & @gennolo, the answer is so obvious!

I've added nginx to psacln group, nginx & apache services restarted and the problem persists.

 

[UFHH01]

Hi Polsys,
Hi gennolo,

if you followed the suggestion, to "chown" the files/folders as described at => #14 <= then you have to check your ".htaccess" - file and the corresponding (additional) nginx directives for possible misconfigurations.
Reconfigure possible misconfiguration with the command "/usr/local/psa/admin/sbin/httpdmng --reconfigure-all".

Try as well to restart apache, nginx and the depending php-fpm service for that domain/subdomain ( example for Ubuntu/Debian and Plesk-PHP-5.6 : "service apache2 restart && service nginx restart && service plesk-php56-fpm restart" ).

Consider as well to reproduce the issue once again, after you successfully modified the permissions as suggested, with a new "test_x.jpg" over your Wordpress-Upload-Ajax-page, for further investigations.
Pls. keep in mind, that each change can result in a change of your issue, so old log - entries are no longer valid and can't be used for an actual investigation process.

I made a test in other server with identical Plesk version and SO, in a different infrastructure, and this problem does not occur.

Have another look at this statement from "Polsys". Even that he didn't mention the difference of the infrastructure, you can see, that Plesk/wordpress and the standard settings can't be the root cause of your issue... it has to be something, that YOU changed in past. Try to remember your changes, re-change it to the standards and see, if this changed the described behaviour.

Consider as well to use another log - level ( temporary set up "debug" - level for investigations, pls. ), to get more details in your depending apache/nginx - log - files.

Try to switch off ALL additional plugins and afterwards switch them on, one-by-one, test again another upload, to see if a plugin causes your issue.

 

[Polsys]

Hi @gennolo and @UFHH01, some news about this.

Last evening I launched a new cloud server with a fresh Plesk 12.5.30 ‪and CentOS 6.8. All 100% from scratch with the default configurations, without WordPress. A programmer from my team has hand-written a tiny PHP script to upload files to the vhost. The files are uploaded to the /tmp directory and moved or copied to the vhost final folder, depending of the script configuration. This is what has happened:

• Test 1: The script uploads the file to /tmp and MOVES it to /vhost-root/httpdocs/ -> nginx 403 error
• Test 2: The script uploads the file to /tmp and COPIES it to /vhost-root/httpdocs/ -> NO 403 error / ALL OK

He has posted the question in Stackoverflow: http://stackoverflow.com/questions/...-unaccessible-403-forbidden-files-under-nginx

Just another light over the problem from our investigations. I hope it helps to catch the solution