Resolved After apt-get upgrade: all pages show NGINX permission denied 403/502

[KaiE]

Hi together

I'm running Plesk Onyx on a Ubuntu server, latest version, latest updates. And here the problem starts: since last night after updating some suggested system packages (8 or 10, have to doublecheck which ones), apache and nginx behave strange. Most of the webpages can be accessed but do not show any graphics, some are not even accassible. Errors are 403 (trying to load the graphic) or 502 if the whole page is not accessible.

Logs in proxy_error_log showing clear things:
[error] 24701#1 "/var/www/vhosts/xyz.ch/httpdocs/index.html" is forbidden (13) Permission denied

Has somebody similar effects? I checked all file- and folder-permissions, everything is still unchanged since yesterday. I did a google-research and found only some hints based on .htaccess problems and other 'normal' access-problems, but nothing around this kind of stuff. I changed the PHP settings for one of the domains from "PHP by NGINX" to "PHP by apache" (and vice versa) and it brought back the inaccessable pages - but now they show no more graphics as well.

Today in the morning I did an apt-get udate upgrade and got some new PHP Updates, but no change, still the permission denied-Problem. And yes, the obvious reboot helped either... And yes, I searched the forum as well, some similar problems but not the same (SSL-handshake problems and Centos firewall stuff...)

Any suggestions? I can provide more logs and details later today if I'm home again - no ssh-access from my office..

Thanks for any Ideas!
Kai

 

[thinkingcap]

You need to comb through logs, also run a "nginx -t" to test the Nginx config, "apachectl configtest" to test Apache config.

 

[KaiE]

Hi @thinkingcap

Thanks, I did the tests, both show "syntax is ok" / "test is successful" / "Syntax OK" - that's what I expected since the services start without warnings or errors.

This seems to be right, since all of the webpages are basically reachable, depending on the php-nginx / php-apache setting. PHP itself is working, all pages are parsed and DBs are accessed, content is visible. But if nginx has to access not a php-file but an other local file, e.g. some graphics, nginx has no access rights. Same behaviour on all webpages.

I'll comb through the logs again in the afternoon, but more than the line I wrote in my first post can not be found, neither int nginx logs nor in apache logs. But I'll keep on searching.

And just to repeat: everything startet after updating the last ~10 packages through plesk! I'll look into the update-logs to find out which ones and post them here.

Thanks for the support
Kai

 

[thinkingcap]

Are you using PHP-FPM?

service php7.0-fpm status

 

[thinkingcap]

Have you done anything with headers to enable HSTS or anything like that?

Have you checked ownership and perms of folders?

 

[KaiE]

Yes, I'm using PHP-FPM in three different Versions, all are effected, but all running fine and status is OK.
Files and folders have (still) the same permissions like yesterday, before I updated some system packages.

I really need to go home to check the update history to find out what changed...

Thanks for the support so far, please let me know if you have more ideas - one night of searching and checking was already done by me, but since I had no success I posted my request here.

 

[UFHH01]

Hi KaiE,

Plesk comes with the Plesk Repair Utility ( => Plesk Repair Utility ), so pls. consider to use this utility, if you experience issues/errors/problems, as it investigates misconfigurations and is as well able to automatically repair the depending configuration files and permissions.

Example commands:

plesk repair fs -vhosts -y -v

or

plesk repair fs example.com -vhosts -y -v

or

plesk repair all -y -v

( Pls. look for MORE options at the official Plesk documentation )​

is forbidden (13) Permission denied

Pls. check as well the following KB - article:

=> Site shows: 403 forbidden error​

 

[KaiE]

Hello together

I'm finally on my computer and could find the apt-get upgrade list which causes the trouble:

Start-Date: 2017-08-06 23:26:29
Upgrade:
linux-libc-dev:amd64 (4.4.0-83.106, 4.4.0-89.112),
libapt-inst2.0:amd64 (1.2.20, 1.2.24),
grub-common:amd64 (2.02~beta2-36ubuntu3.11, 2.02~beta2-36ubuntu3.12),
apt:amd64 (1.2.20, 1.2.24),
libkmod2:amd64 (22-1ubuntu4, 22-1ubuntu5),
binutils:amd64 (2.26.1-1ubuntu1~16.04.3, 2.26.1-1ubuntu1~16.04.4),
libmagickwand-6.q16-2:amd64 (8:6.8.9.9-7ubuntu5.7, 8:6.8.9.9-7ubuntu5.9),
sudo:amd64 (1.8.16-0ubuntu1.4, 1.8.16-0ubuntu1.5),
grub2-common:amd64 (2.02~beta2-36ubuntu3.11, 2.02~beta2-36ubuntu3.12),
libapt-pkg5.0:amd64 (1.2.20, 1.2.24),
grub-pc:amd64 (2.02~beta2-36ubuntu3.11, 2.02~beta2-36ubuntu3.12),
kmod:amd64 (22-1ubuntu4, 22-1ubuntu5),
grub-pc-bin:amd64 (2.02~beta2-36ubuntu3.11, 2.02~beta2-36ubuntu3.12),
ntpdate:amd64 (1:4.2.8p4+dfsg-3ubuntu5.5, 1:4.2.8p4+dfsg-3ubuntu5.6),
apt-utils:amd64 (1.2.20, 1.2.24),
libmagickcore-6.q16-2:amd64 (8:6.8.9.9-7ubuntu5.7, 8:6.8.9.9-7ubuntu5.9),
imagemagick-common:amd64 (8:6.8.9.9-7ubuntu5.7, 8:6.8.9.9-7ubuntu5.9),
apt-transport-https:amd64 (1.2.20, 1.2.24),
apache2-utils:amd64 (2.4.18-2ubuntu3.3, 2.4.18-2ubuntu3.4),
base-files:amd64 (9.4ubuntu4.4, 9.4ubuntu4.5)
End-Date: 2017-08-06 23:26:41​

So one of this packages causes the problemn, I guess it's something around apache2-utils or libmagic.
I'll keep you updated which one was the problem...

@UFHH01 thanks for this idea! I'll try to find the causing packet first and use your way as last chance. I had some unpleasant times with repair-scripts in the past, so let's use this as final solution

Cheers
Kai

 

[UFHH01]

Hi KaiE,

Plesk does not ship one of your listed packages above, so pls. consider to ask related questions at the forum => Home > Forum > General Discussion > Open Topics ... as they are not related to Plesk and Plesk products/components.

and use your way as last chance. I had some unpleasant times with repair-scripts in the past, so let's use this as final solution

Actually, you really should consider to use the Plesk Repair Utility as your FIRST choice, as it is as well able to only to investigate possible issues/errors/problems/. Instead of the "-y" - selector, you would just use "-n" instead, so that no automatic repair would be done, but you still would have the investigation part!

More informations about the Plesk Repair Utility can be read at the official Plesk documentation, already linked above. ​

 

[KaiE]

@UFHH01 well, you are right, none of the packages is shipped by plesk. BUT: they are installed via PLESK WebGUI and results in problems - this might be still interesting in the PLESK Onyx for Linux section, since it is a linux server and makes problems with linux packages... But you might be right and I'll move this thread into the other section later (if I am allowed to do so )

On the other hand, thanks a lot for showing examples how to use the Plesk Repair Tool - and it's always good to know there is a "I'm not touching anything"-mode build in I'll give it a try soon!

At the moment I am upgrading a second, identical, server with exact the same 20 packages. I upgrade package by package and test the function of everything. My main candidat (apache) is the last package I'll update...

 

[UFHH01]

Hi KaiE,

BUT: they are installed via PLESK WebGUI and results in problems

Nope, you are not able to install these packages over the Plesk Control Panel. All these packages are ( basic ) VENDOR software packages, which you need for your server, but which are not related to Plesk itself. Plesk might depend on one of these software packages, but as I already stated, Plesk doesn't ship them.

If you have difficulties to administrate your server and it's software packages, pls. consider to ask for professional services, which Plesk offers as well:

=> Plesk Admin Services​

 

[KaiE]

@UFHH01 umm...sorry, I don't want to contradict with the Plesk Guru, but in my PLESK Onyx 17.5.3 Web Host Edition are this packages installable through Plesk. It's a great feature for every (poor and lazy) sysadmin! And it is not the plesk update page, it's a separate function, i know, and the packages are not provided from Plesk./Odin, that's correct.

Here's a screenshot from the second server, with current available packages. This are obvious packages from plesk, but here the plain apt-get packages are shown as well.

 

[KaiE]

In other news, none of the packages above installed on an identical second server led to the same situation - so it is a different problem, which surprises me a lot.

I'll digg deeper, the second night.

 

[UFHH01]

Hi KaiE,

is there a special reason, why you don't simply log in as "root" over SSH and perform the commands:

aptitude upgrade

followed by

plesk installer --select-product-id plesk --select-release-current --reinstall-patch --install-component base

followed by

plesk repair all -y -v

?

 

[KaiE]

@UFHH01 Yes, there was a reason until now: I really was convinced that this behaviour started exactly after upgrading this packages (and it did, I swear, I can see it in the logs!). And from my experience it often happened that there were broken php-, apache- or plesk packages who broke the services.

But now, after upgrading and downgrading, I'll do it your way - and likely should have done it right from the begining. I keep you updated...

 

[KaiE]

@UFHH01
@thinkingcap

The solution was in the KB article
And it was so obvious - only nginx couldn't access any files and folders, written clearly in the log. Shame on me, this was so basic, I didn't even thought about this, just checked the files and folders permission.

On the other hand: why happened this? It had definitively something to do with this upgrades and some post-upgrade-scripts from plesk. Probably a rare bug which removes the nginx permissions from the system like if nginx would have been removed. Very strange!

But thanks again for all things both of you posted! Every bit helped and I learned a lot about the plesk repair utility and about the possibility that even the most basic things can be the reason for an error.

And I learned, that at least in the world of IT the people help each other, a shame this is more and more rare in the real world.
Good night and good luck!
Greetings from Bern
Kai