Question Site spamming headers, want to find out what it's doing.

[nelteren]

Server operating system version

Debian 13.7

Plesk version and microupdate number

18.0.81.3

Symptom: as in this:

https://support.plesk.com/hc/en-us/articles/12377655871767-Plesk-site-unavailable-502-Bad-Gateway-upstream-sent-too-big-header-while-reading-response-header

I can't get a handle on what's happening behind the scenes here. The site's configured as a FPM application served by nginx. If I raise the header size to a pretty rediculous amount the site can work.

But if I check the actual headers in the site in firefox it's 1.2KB? Almost nothing compared to 128K (growing over time, probably) it "needs" to be set to to not crash in a 502 error. I want to say output the headers it's sending internally to some files for just this domain. How do I do that?

Could someone provide me some simple, human-readable instructions as to how to debug this issue? I just need to prove that the site's doing something silly with the headers like copying the same one a bunch of times, not actually fix it. All the layers of abstraction are hiding the actual issue from me. How do I begin to dig down?

My setting isn't too small, this misbehaving site's nginx/fpm pool looks like it's really actually bugged and sending a huge header in the background somewhere and I want to find out why. How do I do that in this complicated setup?

 

[tessa67]

Hey,

the 1.2 KB in Firefox is a bit of a red herring. That’s what nginx sends to the browser after it’s done. The buffer you’re raising is for what nginx reads from PHP-FPM, and there’s more in there than just headers.

Most likely suspect: PHP stderr output. nginx reads FastCGI stderr into the same buffer as the response headers. If your pool has catch_workers_output = yes and the site throws lots of warnings/notices, you get piles of “PHP message: ...” lines before the headers are complete. nginx then complains “upstream sent too big header” even though the real headers are tiny. That fits your symptoms: a huge buffer fixes it, the browser shows almost nothing, and it keeps growing over time.

Second suspect: really bloated headers (Set-Cookie spam, giant Link header). Less likely, since you’d normally see that in Firefox too.

Check the domain’s error log:

grep -c "sent in stderr" /var/www/vhosts/system/<DOMAIN>/logs/error_log
grep "sent in stderr" /var/www/vhosts/system/<DOMAIN>/logs/error_log | tail -n 3

If you see “FastCGI sent in stderr: PHP message: ...” that’s your answer. Fix the PHP warnings or turn off catch_workers_output in the pool.

Talk to FPM directly, bypassing nginx:
First find the socket:

grep fastcgi_pass /etc/nginx/plesk.conf.d/vhosts/<DOMAIN>.conf

Then ask FPM yourself. --> stdout and stderr go into separate files:

apt install -y libfcgi-bin
SCRIPT_NAME=/index.php SCRIPT_FILENAME=<DOCROOT>/index.php REQUEST_URI=/ QUERY_STRING= REQUEST_METHOD=GET SERVER_NAME=<DOMAIN> HTTP_HOST=<DOMAIN> cgi-fcgi -bind -connect <SOCKET_PATH> > /tmp/fpm-out.txt 2> /tmp/fpm-err.txt

Placeholders: <DOCROOT> is usually /var/www/vhosts/<DOMAIN>/httpdocs, <SOCKET_PATH> is the path from the fastcgi_pass line without the unix: prefix. If the site forces HTTPS, add HTTPS=on to the variables.

Compare sizes:

wc -c /tmp/fpm-err.txt
sed -n '1,/^\r\?$/p' /tmp/fpm-out.txt | wc -c
sed -n '1,/^\r\?$/p' /tmp/fpm-out.txt | cut -d: -f1 | sort | uniq -c | sort -rn | head

Big fpm-err.txt and small headers: it’s the stderr noise.
Small stderr and big headers: the last command shows which header is repeated.

It would be helpful for all, if you could post the output without cookie values.

Regards
Tessa

 

[nelteren]

So I did some more digging, and between some chatbots, some coding, and most importantly some comments squirreled away deep inside a stack overflow discussion, re-reading the php settings thing in Plesk, and realizing some different problem may be related: See this comment:

This answer hit the nail in the head. Sometimes it's not just the configuration of nginx, but what's actually producing the header. When error_reporting contains the notices but display_errors is off in php.ini, all the messages show up in FCGI header instead of content.

My other existing issue is documented here:

Question - How to localize PHP-FPM logs into domain's log folder?

Is there a guide on setting up php-fpm to log to its client log in their log folder? I've had a few clients who had the warning deprecation log flood caused by the actual site's code; nothing php.ini had control over growing into huge multi-GB files. 1) I want these log files in the client's...

talk.plesk.com

I'm still seeing the same thing. There's the nginx error logs in the domain, but that doesn't contain the FPM log. FPM throws everything and the kitchen sink into one giant file at /var/log/plesk-php83-fpm/error.log.*, when what I want is a log file per domain as the starting point. I don't need FPM to forward PHP errors to nginx through a mousehole and nginx then printing them, just for those errors to be accessible to the site owner.

There's this topic here that points to some info: https://serverfault.com/questions/1193464/per-domain-error-log-for-php-fpm-and-apache
But this is for a regular linux server... not one with the quite complicated plesk program stuffed over top. Going "around" plesk to the backend and directly hacking something up is just going to result in a bunch of pain later when things get overwritten/changed by updates or restarts, so I need a way to properly set error logs (iff indeed this is the problem, which I think is pretty likely at this point) in a way that is reasonable.

I think what may be happening is that certain kinds of PHP errors, including warnings related to attempting to access files outside of the site path (wordpress plugins doing stupid things) ... are being sent into the PHP-FPM header buffer somehow.

Which is an incredibly stupid thing to do, as that buffer is positively tiny. Any kind of PHP warning or error that wordpress generates is incredibly verbose, I've seen megabytes be produced by a single request with ease before via 'security plugin' databases spewing banlist data all over traces and what not. Why would it do that? Why not write to a log file and call it a day? Can I set it up to be... not utterly insane, please?

Here's the conditions I'd like to fulfill:

* Yes, I do want per-domain logs. Site owners may actually fix things if they're presented with a slew of errors rather than them getting blackholed out of view.
* No, do not display all the errors and warnings on-screen to all site visitors.
* Standard level I've set is fine (E_ALL & ~E_NOTICE & ~E_STRICT & ~E_DEPRECATED)
* Keep using Nginx/FPM as it's the most modern/fastest way of serving PHP. I'm otherwise happy with this way of organizing the web stack.
* Don't put the error log for a request in a 32 KILOBYTE buffer. No, my elephant of an error can't fit through that mousehole.
* Do it via plesk or plesk cli so plesk won't ruin the setup with the next update.
* An explanation as to why and how it works rather than cargo-cult copying working but fragile settings is more useful for people in the future.

As the problem is squirrelled away somewhere in the closed-encrypted-source thing that's generating PHP settings files with wrong log configurations I need some help with this. To do, according to the quote;

So I believe the correct answer is: fix your fpm config so it logs errors to disk.

I found where plesk actually stores the output file I want changed, in /opt/plesk/php/8.x/etc/php-fpm.d/. These files look like (red.)

; ATTENTION!
;
; DO NOT MODIFY THIS FILE BECAUSE IT WAS GENERATED AUTOMATICALLY,
; SO ALL YOUR CHANGES WILL BE LOST THE NEXT TIME THE FILE IS GENERATED.

; If you need to customize this file, use either custom PHP settings tab in
; Panel or override settings in /var/www/vhosts/system/ims.acme.com/conf/php.ini.
; To override pool configuration options, specify them in [php-fpm-pool-settings]
; section of /var/www/vhosts/system/ims.acme.com/conf/php.ini file.

[ims.acme.com]
; Don't override following options, they are relied upon by Plesk internally
prefix = /var/www/vhosts/system/$pool
user = bressernu
group = psacln

listen = php-fpm.sock
listen.owner = root
listen.group = psaserv
listen.mode = 0660

; Following options can be overridden
chdir = /

; If php_value[error_log] is not defined, worker errors will be sent to
; web server and main php-fpm error log
catch_workers_output = yes

; php.ini custom configuration directives
php_admin_value[disable_functions] = "opcache_get_status"
php_value[error_reporting] = 22519
php_admin_value[max_execution_time] = 30
php_admin_value[max_input_time] = 60
php_admin_value[memory_limit] = 128M
php_value[open_basedir] = "/var/www/vhosts/acme.com/:/tmp/"
php_admin_value[post_max_size] = 8M
php_admin_value[upload_max_filesize] = 8M

; Following directives define pool configuration
pm = ondemand
pm.max_children = 5
pm.max_requests = 0
pm.max_spare_servers = 1
pm.min_spare_servers = 1
pm.process_idle_timeout = 10s
pm.start_servers = 1

The problem is there's no value generated there for the log file. There's specifically a missing generated line that should look like this;

php_admin_value[error_log] = "/var/www/vhosts/acme.com/logs/ims.acme.com/php-error.log"

Without that, it's trying to put it through a buffer that was never created to handle that much data. Just a few cookies.

Now the main question: After I think I've figured out the cause... How do I get that in there? Why's plesk not putting that in there? And also stop the existing log path forwarding it to nginx, I'm fine with the nginx error log only having nginx errors and the php(fpm) errors in two locations rather than being 4-5x duplicated; one in /var/www/vhosts/system/$hostname/logs, and the other one a symlink there.

There should be two levels though as I don't want the site owner being able to alter the logs I see as admin. Just to view them from within Plesk.

 

[nelteren]

I believe that adding the following to the custom settings in each site for php-fpm (under php-settings) resolves the issue:

error_log = "{WEBSPACEROOT}/logs/ims.acme.com/error_log"

(Note, this is an example for a subdomain, with a regular domain, it would be: )

error_log = "{WEBSPACEROOT}/logs/error_log"

After this, check the chown settings for the file, then OK to restart the php-fpm pool and I see the logs going into those files now. (Which hopefully means no longer being passed over FPM if I'm understanding the stack overflow threads correctly). Will test tomorrow if I no longer need those additional nginx directives.

 

[tessa67]

Nice digging, you’re on the right track. Small correction to the quote: It’s not literally the HTTP headers, it’s FPM’s stderr stream, which nginx reads into the same buffer. Same effect, though.

The pool file you found already tells you the way out: it’s generated, but the header says to use the [php-fpm-pool-settings] section in the domain’s php.ini. That survives regeneration.

Pls edit the domain’s php.ini:
(/var/www/vhosts/system/<DOMAIN>/conf/php.ini) or put the same lines into “Additional configuration directives” under the domain’s PHP settings in Plesk:

[PHP]
log_errors = On
display_errors = Off
error_log = <LOGFILE>

[php-fpm-pool-settings]
catch_workers_output = no

What it does:
error_log makes PHP write errors straight to a file instead of stderr. catch_workers_output = no stops FPM from forwarding worker output to nginx, so nothing is squeezed through the buffer anymore. display_errors = Off keeps errors away from visitors.

Watch the path:
The worker runs as the domain’s system user, so it must be able to write to <LOGFILE>. The Plesk logs folder is normally root-owned, so writing there will probably fail. Create a writable folder instead and test it:

mkdir -p /var/www/vhosts/<DOMAIN>/php-logs
chown <SYSUSER>:psacln /var/www/vhosts/<DOMAIN>/php-logs

Then use /var/www/vhosts/<DOMAIN>/php-logs/php-error.log as <LOGFILE>. I’m not sure Plesk’s own log rotation covers that folder, so check that, otherwise your multi-GB problem just moves.

Apply and verify:

plesk bin php_handler --reread
systemctl reload plesk-php83-fpm
grep -E "catch_workers_output|error_log" /opt/plesk/php/8.3/etc/php-fpm.d/<DOMAIN>.conf

If the generated pool file now shows the changed values (or the same via the UI), it’s applied. Then load the site and see whether the 502 is gone and the log fills up.

Admin vs. site owner:
Plesk lets you hide the PHP settings from the customer (via the subscription’s permissions). Whoever can edit that php.ini can change the log path, so that’s the lever for your two-level setup. I haven’t tested whether php_admin_value[error_log] in the pool section also locks it against ini_set, so try it on a test domain first.

Honest caveat: I haven’t verified every detail on a live Plesk 18.0.81, so run it on one domain before rolling it out.

Regards
Tessa

 

[tessa67]

I believe that adding the following to the custom settings in each site for php-fpm (under php-settings) resolves the issue:

error_log = "{WEBSPACEROOT}/logs/ims.acme.com/error_log"

(Note, this is an example for a subdomain, with a regular domain, it would be: )

error_log = "{WEBSPACEROOT}/logs/error_log"

After this, check the chown settings for the file, then OK to restart the php-fpm pool and I see the logs going into those files now. (Which hopefully means no longer being passed over FPM if I'm understanding the stack overflow threads correctly). Will test tomorrow if I no longer need those additional nginx directives.

Glad that worked, and yes, that matches the comment in the generated pool file: if error_log isn’t set, worker output goes to the web server, otherwise PHP writes straight to the file.

Two things I’d double-check:

File name:
{WEBSPACEROOT}/logs/error_log is the same name Plesk uses for the web server’s own error log in that folder, so you may end up mixing two writers into one file, and your chown could affect the web server’s log. I’d use a separate name:

error_log = "{WEBSPACEROOT}/logs/php_error.log"

Ownership after rotation:
The file has to be writable by the domain’s system user. If Plesk’s log rotation (or anything else) recreates it as root, PHP can’t write anymore and you may be back at the old behaviour. Check after the next rotation, and see whether the file is rotated at all, otherwise you just move the multi-GB problem.

To verify tomorrow:

ls -l /var/www/vhosts/system/<DOMAIN>/logs/
tail -n 20 /var/www/vhosts/system/<DOMAIN>/logs/php_error.log

If the 502 is gone and the file fills up, you can drop the extra nginx buffer directives.

Regards
Tessa