ISSUE - Nginx config generation failure and http2 directive implementation failure

[trialotto]

Username:

TITLE

ISSUE - Nginx config generation failure and http2 directive implementation failure

PRODUCT, VERSION, OPERATING SYSTEM, ARCHITECTURE

Tested on :

Product version: Plesk Obsidian 18.0.61.4
OS version: Ubuntu 20.04 x86_64
Build date: 2024/05/28 05:00
Revision: 81fad59b98465445debc4a7e2cbef11dc90f99ff

nginx version: nginx/1.26.0

PROBLEM DESCRIPTION

Nginx sends error notification by email for each and every domain on the server :

nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/plesk.conf.d/..


This error notification implies improper usage of the http2 directive, introduced in Nginx 1.25.1


The nginxDomainVirtualHost.php template used for generation of Nginx templates contains

listen <?php echo $OPT['ipAddress']->escapedAddress . ':' . $OPT['frontendPort'] . ($OPT['ssl'] ? ' ssl' : '') ?>;
<?php if ($OPT['ssl'] && $OPT['http2']) : ?>
http2 on;
<?php endif; ?>


The aforementioned code is correct and should have resulted in PROPER REGENERATION of Nginx config.


However, the Nginx config files of each and every domain exhibit

listen [ip address]:443 ssl http2;


In short, Nginx templates are NOT GENERATED with the (new) nginxDomainVirtualHost.php template!!

STEPS TO REPRODUCE

A - STR 1

1 - install older version of Plesk with Nginx version lower than 1.25.1
2 - update Plesk, in order to have Nginx version 1.25.1 or higher, with associated Nginx config templates
3 - check result with nginx -t

B - STR 2

On the test machine, some custom Nginx templates were added to improve the default Nginx caching templates.

It might be the case that custom Nginx templates are interfering with Nginx config generation, since this "generation issue" has been occurring before - it is not limited to http2 directive changes alone.

1 - install older version of Plesk with Nginx version lower than 1.25.1
2 - add custom Nginx templates in /opt/psa/admin/conf/custom/ ....
3 - update Plesk, in order to have Nginx version 1.25.1 or higher, with associated Nginx config templates
4 - check result with nginx -t

ACTUAL RESULT

Plesk Nginx config files still contain

listen [ip address]:443 ssl http2;

even if the new nginxDomainVirtualHost.php generation template is present!

EXPECTED RESULT

The new nginxDomainVirtualHost.php generation template should generate

listen [ip address]:443 ssl;
http2 on;

ANY ADDITIONAL INFORMATION

The current issue is a bug that has been present before, so it is inherent to the process of Nginx config regeneration.

It might be related to the presence of custom Nginx config templates or, better said, the lack of dealing with them.

It might also be related to systemic behavior of Nginx - in this case, this would be very very very unlikely.

It can also be related to the chronological order in which

- Nginx updates,
- Nginx config generation templates,
- Nginx config generation,

are installed and/or executed.

It would be my educated guess that the "choronological order" of process is the root cause of the problem!

After all, regeneration of config files after updates often does the trick!

YOUR EXPECTATIONS FROM PLESK SERVICE TEAM

Confirm bug

 

[trialotto]

@Plesk Team,

In addition, it has to be duly noted that the Webserver Configurations Troubleshooter extension does not work properly.

A run of the Webserver Configuration Checker - surprisingly - results in :

Checking Plesk version
Installed Plesk version/build: 18.0.61 Ubuntu 20.04 1800240528.05

Checking Apache configuration ....................................... [OK]

Checking for custom configuration templates
Custom template /opt/psa/admin/conf/templates/custom/domain/service/nginxCachePath.php has been found
Custom template /opt/psa/admin/conf/templates/custom/domain/service/nginxCacheProxy.php has been found

Some custom configuration templates have been found. The custom
templates have higher priority than default templates in case of
configs generation.
Please check documentation for details:
https://docs.plesk.com/current/redi...istration-guide-linux&page=68693.htm[WARNING]

Checking associations between domains and IP addresses .............. [OK]

Checking for corrupted reference between IP collections and IP
addresses ........................................................... [OK]

Checking for links between APS applications and subscriptions ....... [OK]

Checking for nginx ULIMIT value ..................................... [OK]

Checking for extra configurations in database not owned by any object
................................................................... [OK]

Found errors: 0; Found Warnings: 1


All of the above indicates :

1 - that all is fine, except for 1 warning : the presence of custom templates

2 - that the Webserver Configurations Troubleshooter will NOT work or will NOT work PROPERLY, in the presence of custom templates

Please note that "not work" or "not work properly" is an issue of "false positives" - the Webserver Configurations Troubleshooter indicates that certain config files are fine, but they are actually not.

This issue of "false positives" is something that needs to be addressed.

However, there might be a relation between the bug reported and the "issue of false positives".


Kind regards....

 

[Kaspar@Plesk]

Thank you for your report. We are aware of the issue and it will be fixed in future updates. Bug can be tracked with ID PPPM-14442 on the change log.

There is a support article detailing a workaround:

https://support.plesk.com/hc/en-us/articles/23555281297431-nginx-warning-after-installing-Plesk-update-61-warn-the-listen-http2-directive-is-deprecated

 

[trialotto]

Thank you for your report. We are aware of the issue and it will be fixed in future updates. Bug can be tracked with ID PPPM-14442 on the change log.

There is a support article detailing a workaround:

https://support.plesk.com/hc/en-us/articles/23555281297431-nginx-warning-after-installing-Plesk-update-61-warn-the-listen-http2-directive-is-deprecated

@Kaspar@Plesk

To be precise, the bug reported is not the same as the bug with ID PPPM-14442.

Nevertheless, a bug fix can tackle both problems.


The bug with ID PPPM-14442 is a mix of various issues related to http/3 support by Plesk and/or Nginx and the various (root) causes of these issues.

In essence, the aforementioned bug is a more complex bug, probably not to be resolved quickly or in a straightforward manner.


However, the reported bug is different by nature - there is no complex relation with other issues or causes, all the "goodies" are in place.

Stated differently, Plesk does not generate the proper Nginx config in specific circumstances, despite of the presence of the good Nginx config template.

A quick and straightforward solution should be present, given the fact that the proper Nginx config templates are in place.


I would not wait to combine a solution for the reported bug and the bug with ID PPPM-14442.

The reported bug makes Nginx vulnerable to (very advanced) attacks, which are known to exist when the "http2 on" directive is missing and Nginx config is very basic (which is more or less the case with the default Nginx config from Plesk).

It is just some food for thought.


Kind regards....


EDIT : it is also not the best way to suggest the KB article. Sure, this is a workaround, since it should trigger regeneration of config files. However, it might be very confusing to tell people to run a command related to http/3 if and when they have a http/2 related config issue. In essence, the workaround might result in people activating http/3 and hence triggering the http/3 related issues that are not solved yet (and that are can be deemed very major issues).

 

[Kaspar@Plesk]

I am pretty sure your bug report describes the same issue with the Nginx http2 directive as the one we have filed with ID PPPM-14442.

 

[trialotto]

I am pretty sure your bug report describes the same issue with the Nginx http2 directive as the one we have filed with ID PPPM-14442.

@Kaspar@Plesk

It really is not.

The workaround solution only fits the bill, because it causes Nginx config templates to be regenerated.

However, the reported bug is related to the inability of Plesk config generators to take into account custom Nginx templates.

That inability has been present up to date ...... in previous releases of Plesk, it was also present.

Proof of the former statement can also be found in the fact that the Webserver Configurations Troubleshooter does give a notification when doing a check in the presence of custom templates : it is reported as "higher priority" (assigned to these custom templates), but the check of Nginx config templates fails and/or is not executed.

One can find additional proof by simply removing the custom templates and running the Webserver Configurations Troubleshooter (without custom templates) and then one gets the "OK" - implying that the presence of custom templates is not properly dealt with by the "Troubleshooter".

Since the "Troubleshooter" is more or less the Plesk GUI of (part of) the Plesk CLI Repair utility, this issue also applies to the Plesk CLI Repair Utility.

Since the Plesk CLI Repair Utility is based upon a subset of CLI commands, this issue finds its root cause of the problem in one of these commands.


Feel free to disagree, I am fine with that.

However, the presence of custom Nginx config templates has been causing issues with

- the Webserver Configurations Troubleshooter
- the Plesk CLI Repair Utility
- some Plesk CLI commands

for a long time now, in some cases even causing major issues (for instance, in the past, the Webserver Configurations Troubleshooter did not work at all).


It is fine to focus on a solution dealing with HTTP/3 related issues, but that will cause additional problems.

Some of these problems have already popped up on Plesk Forum, others have not surfaced yet.

In essence, Nginx did not patch some of the HTTP/3 related issues in the packages that are underlying the Plesk Nginx packages.

Stated differently, workarounds for the HTTP/3 related issues might solve the reported bug, but they are not the final solution.


In my humble opinion, it can only be recommended to separate the reported bug from the HTTP/3 related issues.

After all, when fixing the HTTP/3 related issues (if possible!), then one would still have the inherent issue of the reported bug (read: custom Nginx config templates are not properly dealt with) being present in Plesk.


Sure, I can imagine that you ask by now "why bother?"

Well, there is a simple explanation : default Nginx config, as provided by Plesk, is sufficient in a general sense, but NOT in most use cases.

For instance, Nginx based caching really requires custom Nginx templates that are (on the one hand) filling the gaps and voids in the default Nginx caching config provided by Plesk and (on the other hand) optimizing the caching config for the specific use case.

The Nginx caching config is also something that has been on the Plesk roadmap for many years now, to no avail up to date.


In summary, at this moment there is a necessity (read: HTTP/3 related issues) and an opportunity to get Nginx config improved.

I am fine with whatever Plesk Team decides to do, but I can only recommend to separate the reported bug from the HTTP/3 related issues.


Kind regards....

PS You can take my word for it (or not), but the current compilation of Nginx packages is a "too early" adoption of HTTP/3 support.

 

[Kaspar@Plesk]

It really is not.
[...]

However, the reported bug is related to the inability of Plesk config generators to take into account custom Nginx templates.

[...]

It seems I misunderstood your bug report. However, if your report is not about the new Nginx http2 directive causing warnings on Plesk .61, than I must admit I am not really sure what to make of your report. At current I am not sure how your current report relates to Nginx templates. I am afraid I need some help from you here.

Are there any other STR and alternative "Actual" and "Expect" results that can be used to better preproduce the issue?

 

[trialotto]

@Kaspar@Plesk

The question

Are there any other STR and alternative "Actual" and "Expect" results that can be used to better preproduce the issue?

can be answered with a definite "no".


Although the reported bug is very likely to be associated with one of the CLI utilities underlying

- the Webserver Configurations Troubleshooter
- the Plesk CLI Repair Utility

the reported bug seems to be only manifesting itself when updates and/or micro-updates are (explicitly read: have been) executed.


Hence the suggested STR and the preliminary conclusion that it might be caused by the chronological order processes are executed at update time.


Stated differently, if

- the proper Nginx config templates are present after an update or micro-update, (and)
- the proper Nginx config is not generated after an update or micro-update, (and)
- the proper Nginx config can be (manually) generated after an update or micro-update,

then something has gone wrong during the update process and the inherent config generation at update time.

The only explanation for that (unexpected) behavior can be, given the fact that post-update manual config generation succeeds, that

1 - the chronological order of processes is impeding proper Nginx config generation, (AND/OR)

2 - the CLI utility used during updates is not properly dealing with custom config templates.


By the way, it is also safe to say that the whole PHP based generation process for config templates is a bit "dirty", certainly for Nginx config.

In my humble opinion, Nginx config and certainly Nginx config customization should be done by means of the "include" directive.

Nginx by default allows for config customizations in separate config files, which customizations can be included with the "include" directive (explicit method) OR which customizations are read by Nginx natively (implicit method - Nginx does that automatically).

Plesk by default does not have the "include" directive in (most of) the Nginx config templates.

However, even though Plesk does use the right approach to not use the "include" directive (explicit method) in (most) default Nginx config templates, there are many use cases in which explicit calls for Nginx are desirable or even necessary.

For instance, consider Nginx based caching config - this should be placed in the appropriate location block : the / location block in Plesk.

Thus, for Nginx based caching one could want a custom Nginx config template that generates config by means of a PHP based generation process.

Nevertheless, it is not a necessity ........ and it might be even a burden, since "general" Nginx caching optimization will be the same for all (cached) domains across the entire server : only a specific subset of variables (read: variables that should not be cached) will differ per domain.

Plesk wants to use a PHP based generation process for Nginx templates, but it is not the best choice for specific customizations.

Plesk did not change this historical choice and, therefore, an issue with custom Nginx config generation persists for years now.


In short, Plesk Team should have a good look at

1 - the chronological order of processes run at update time and their respective effects on generation of Nginx config templates,

2 - the CLI utility (or other code) that causes the Nginx config generation to be flawed in the presence of custom Nginx config templates,

3 - a different approach to include Nginx config customizations - it would solve a lot of problems.


Kind regards....

PS An increase in the number of "include" directives is increasing the attack surface to some extent, but this is another story. The risk can be controlled fairly easily and the separation of custom config from PHP based generation of "regular" config files will increase performance of tools like the Webserver Configurations Troubleshooter and/or the Plesk Repair CLI utility - they can be quite slow.

 

[nmdpa3]

The referenced support article fix, does not seem to work.

 

[nmdpa3]

The referenced support article fix, does not seem to work.

Was a config issue on my side...disregard.

 

[trialotto]

It seems I misunderstood your bug report. However, if your report is not about the new Nginx http2 directive causing warnings on Plesk .61, than I must admit I am not really sure what to make of your report. At current I am not sure how your current report relates to Nginx templates. I am afraid I need some help from you here.

Are there any other STR and alternative "Actual" and "Expect" results that can be used to better preproduce the issue?

@Kaspar@Plesk

ISSUE REOCCURS on updated servers.

Updates are from Plesk Obsidian 18.0.61.x to Plesk Obsidian 18.0.61.5

Can this issue be resolved before the next MU?

Kind regards....

 

[Kaspar@Plesk]

It's on our todo list, however I have no ETA on when issue PPPM-14442 wil be fixed.

 

[MARS_NETWORKS]

Stil no having an ETA ?
It chrashed one of our servers
We had to turn on apache only...

 

[Sebahat.hadzhi]

After a further consideration PPPM-14442 was not recognized as bug due to low business impact (no actual downtime caused on websites after update) and automatic reconfiguration of all domains is not worth it as it temporarily brings the sites down. Did apply the workaround described here not solve the issue?

 

[Bitpalast]

Would not a solution ideal that runs a regex replace operation on the webserver configuration files so that upon the next reload or restart the updated version is loaded? It would avoid the need for a short downtime due to reconfiguration of all files by the normal Plesk reconfiguration tool.