Resolved Debugging "Premature end of script headers"

[SKDamon]

Hey everyone,

I experience weird behavior on the domains of 1 customer I have in my Plesk Panel.
Generally, sites take a tad long to load. >5 Seconds for a Wordpress index.
But occasionally, ony 500 Server Error is returned.
The Apache error log contains this:

[warn] mod_fcgid: read data timeout in 60 seconds
[error] Premature end of script headers: index.php

I followed numerous guides, increasing PHP memory and MySQL memory, without success...

Now, how can I find out what is causing this error? I think it's a load or DB issue, since other customers don't experience the problem. The error occured 1-2 Times in other customers' logs. On the 1 affected customer, however, it shows several times an hour, quite randomly. I never triggered it on PHP files without any DB Access. Also, File Permissions are OK, since the error occurs randomly and the majority of requests succeed.

the suexec.log is rather large, but only has lines stating uid/gid and cmd, so I think those aren't errors.
Running PHP from CLI also takes quite long to return the Page HTML.

Are there any Plesk logs from which I can see if the User ran into some quota limit?
Is there any additional source of information which might help me?

Some more system info:
CPU Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (12 core(s))
Version Plesk v12.5.30_build1205150826.19 os_Ubuntu 12.04
OS Ubuntu 12.04.5 LTS
RAM 64GB
PHP 5.6.24 runs via FCGI

BTW No, I didn't increase timeout values. I want the problem fixed, that beast of a server should really be able to deliver a WP Blog in less than 60 seconds

 

[SKDamon]

New Theory:
That 1 affected client runs WordPress.
1st Appearance of the error is [Tue Aug 09 09:39:47 2016].
Apparently, 3 hours before plesk-wp-cli was updated: 2016-08-09 06:29:42 status installed plesk-wp-cli 0.24.0-ubuntu12.04.16080218
There were requests to the blog inbetween wp upgrade and appearance error. Also, other domains from the same user which don't run WP are affected.

In DEBUG mode, WP spits out

Warning: mysqli_real_escape_string() expects parameter 2 to be string

on EVERY request. mysql logs don't show query errors, also the site displays fine (no HTTP error codes).

Any ideas what the hell is going on?
Thanks, all help is MUCH appreciated!

 

[UFHH01]

Hi SKDamon,

first, pls. be informed that this is a "Plesk related" - forum. Consider to ask your question(s) with wordpress - related issues/problems/failures at for example "https://wordpress.org/support/".

However... just a few hints, how to investigate wordpress - issues/problems/failures:

• Increase your log - level for the domain and use for example temporary the "debug mode", in order to get a more decent output in your logs, which often points directly to issues. It might as well help to follow: => https://codex.wordpress.org/Debugging_in_WordPress

... or just temporary modify your "wp-config.php" with:

define('WP_DEBUG', true);
define('WP_DEBUG_LOG', true);
define('WP_DEBUG_DISPLAY', true);

• Switch off wordpress - plugins one-by-one and test if a plugin is the root cause of your issue.

• Use another ( standard ) theme and test, if your described issue(s) appears here as well.

• Check your folders for the correct permissions.

"plesk-wp-cli" for example has got nothing to do with your wordpress site itself and your described issue(s). It is a set of command-line tools to manage wordpress - installations. Only in very, very, very rare cases, the root cause of your issue(s) can be found here.

In DEBUG mode, WP spits out

Warning: mysqli_real_escape_string() expects parameter 2 to be string

Consider to add the previous request for investigations, because nobody could even guess the request, which caused a MySQL - error.


If you would like people to help you with investigation(s), pls. include MORE details ( which theme you use, which plugin(s) you use, did you try to switch the PHP-handler - if yes, pls. specify, ... ) The more details you provide, the better will be the help/suggestion(s), but pls. remember again, that this forum is... yes... I mentioned it before... a Plesk-related forum.

 

[SKDamon]

Thanks for your answer!

Indeed, I just wanted to make sure the problem is(n't) caused by the server system (CPU/MEM limit by plesk, wrong config...).
Just to be on the safe side: Can a WP update really cause this intermittent behavior?
The errors are on every page delivered. But only some percentage of requests result in a 500 error.
Which is why I think the problem is dependent on load.

Can you think of any other reason why the problem only (intermittently) occurs on this one customer's domains?
(All of them, even the ones not running WordPress)

 

[UFHH01]

Hi SKDamon,

Indeed, I just wanted to make sure the problem is(n't) caused by the server system (CPU/MEM limit by plesk, wrong config...).

You don't provide enough information for possible investigations, so an answer to your question can only be: "Plesk standard settings and modifications don't cause such issues/problems/failures".

Just to be on the safe side: Can a WP update really cause this intermittent behavior?

Again, you don't provide enough informations for a decent answer to your question and investigations can't be done without them. We would be able to guess hundreds of reasons, but all of which may not apply to your environment and it's modification and configuration.

Can you think of any other reason why the problem only (intermittently) occurs on this one customer's domains?

False htaccess - entries, misconfigured/missing nginx rewrites rules, bad coding ( theme(s) / plugin(s) / wordpress itself )... there are hundreds of possible causes, which could lead to issues/failures/problems.

 

[SKDamon]

"Plesk standard settings and modifications don't cause such issues/problems/failures".

Thanks, that's all I wanted to hear (/read).

I was actually in the middle to write a close-up for this thread, as the problem was resolved.
See my next post for more information
Thanks for your support!

 

[SKDamon]

Hey,

Their sites were hacked (probably through old WP and/or plugins).
Fulll disclosure for anyone experiencing bad performance / weird PHP errors on all of one customer's domains:

I searched the logs for the first appearance of the error, which was Aug 9 9:37

As UFHH01 kindly told me, the only packet on my system upgraded that day, plesk-wp-cli shouldn't be related to my problem.
So I went to look at the actual files on the web server. Perhaps the client changed something that morning and forgot(tm) to tell us?
Nope, most recent changes were before Aug 9th...
But ls -t showed me some file was modified Aug 9 2014 - 9:37
And I'm pretty sure mera.php doesn't belong in /wp-includes...

Yep, it's malicious:
http://stackoverflow.com/questions/39012433/wordpress-site-hacked-suspicious-php-file

Bash-foo to the rescue!
First, I thought only that WP instance was affected, so I downloaded a clean tarball, extracted it and diff'ed all files to the ones present on my server:

find . -type f | xargs -i md5sum {} > /tmp/md5_stock.txt
find /var/www/vhosts/affected_user/httpdocs/domain/ -type f | xargs -i md5sum {} > /tmp/md5_installed.txt
diff md5_stock.txt md5_installed.txt | grep php | sed -e '/.*wp-content\/plugins/d' | sed -e '/.*wp-content\/themes/d'

wp-includes/meta.php had a different checksum, so a diff between the installed and clean version returned added malicious code.

Now to search for more infected files:

find /var/www/vhosts/affected_user/ -regex ".*\.php" | xargs -i grep -Hin 'file_get_contents("http'

That returned all (11) files infected by that attack. With that info, I was able to remove/restore original state of all those files.
After a reboot for good measure, sites now load OK, logs don't show any errors after the malicious files were cleaned.

Again, Thanks @UFHH01 - if it wasn't for you, I'd still be searching my server's config files for errors or options to tweak.


tl;dr Tell your customers they really need those security patches for their WordPress blogs.