Issue php-cgi segfault

[nisamudeen97]

Hi,

Our plesk server with 64GB ram and Intel(R) Xeon(R) CPU D-1520 with 4 core is showing very poor performance. We are running around 100 live websites in the server. We have done server tweaking for maximum performance.

We are experiencing slowness with websites in the server. We have done detailed investigation of server logs and noticed the "php-cgi segfault" logged in kernal logs. As per our current memory usage we have only 50% usage of ram.

We are also using software raid and its arrays are 100% good status. I would like to have expert advise for fixing the issue.


free -m
total used free shared buffers cached
Mem: 64385 29737 34648 7438 168 20664
-/+ buffers/cache: 8903 55481
Swap: 1021 52 969



Apr 3 11:40:32 ns1 kernel: php-cgi[28623]: segfault at 271f2f0 ip 000000000271f2f0 sp 00007327e808b518 error 15
Apr 3 11:55:32 ns1 kernel: php-cgi[4087]: segfault at 15dfaa0 ip 00000000015dfaa0 sp 000074ad36b91388 error 15
Apr 3 12:00:32 ns1 kernel: php-cgi[6811]: segfault at 3ed7c10 ip 0000000003ed7c10 sp 00007854e7400a08 error 15
Apr 3 12:05:32 ns1 kernel: php-cgi[9923]: segfault at 129dbb0 ip 000000000129dbb0 sp 0000710d53e50478 error 15
Apr 3 12:10:31 ns1 kernel: php-cgi[12510]: segfault at 372d8e0 ip 000000000372d8e0 sp 00007fbfb6bef5f8 error 15
Apr 3 12:15:41 ns1 kernel: php-cgi[15159]: segfault at 2a62840 ip 0000000002a62840 sp 0000744246f89db8 error 15
Apr 3 12:20:45 ns1 kernel: php-cgi[17699]: segfault at 4706070 ip 0000000004706070 sp 0000720658146c28 error 15
Apr 3 12:25:32 ns1 kernel: php-cgi[20243]: segfault at 7cb1be68e8f0 ip 00007cb1be68e8f0 sp 00007f0d5e89d108 error 15
Apr 3 12:30:58 ns1 kernel: php-cgi[23230]: segfault at 6d9350f778f0 ip 00006d9350f778f0 sp 0000722cc2d249f8 error 15
Apr 3 12:35:32 ns1 kernel: php-cgi[25815]: segfault at 73ff6c6fd8f0 ip 000073ff6c6fd8f0 sp 000078c39a81f8f8 error 15
Apr 3 12:40:46 ns1 kernel: php-cgi[28731]: segfault at 757724a908f0 ip 0000757724a908f0 sp 00007d0fac131ff8 error 15
Apr 3 12:45:55 ns1 kernel: php-cgi[32120]: segfault at 6c55203638f0 ip 00006c55203638f0 sp 00007a4309807628 error 15
Apr 3 12:50:55 ns1 kernel: php-cgi[2818]: segfault at 7a8bc0e9d8f0 ip 00007a8bc0e9d8f0 sp 00007f8ed3adcd38 error 15
Apr 3 13:28:55 ns1 kernel: php-cgi[22198]: segfault at 45cade8 ip 00000000045cade8 sp 00007679d5613f78 error 15
Apr 3 13:32:51 ns1 kernel: php-cgi[24669]: segfault at 70da800fa3e0 ip 000070da800fa3e0 sp 000079f790a97238 error 15
Apr 3 18:00:53 ns1 kernel: php-cgi[2230]: segfault at 2ac6580 ip 0000000002ac6580 sp 000071fadce77858 error 15
Apr 3 18:15:57 ns1 kernel: php-cgi[10405]: segfault at 43b2758 ip 00000000043b2758 sp 000079b995d57e58 error 15
Apr 3 22:40:31 ns1 kernel: php-cgi[28851]: segfault at 40ab898 ip 00000000040ab898 sp 00007026ae736f58 error 15
Apr 3 22:49:00 ns1 kernel: php-cgi[950]: segfault at 2ae4f38 ip 0000000002ae4f38 sp 0000742a9ba6d7c8 error 15
Apr 3 23:04:43 ns1 kernel: php-cgi[10181]: segfault at 3dcda98 ip 0000000003dcda98 sp 000073860c09e758 error 15
Apr 3 23:20:47 ns1 kernel: php-cgi[20222]: segfault at 739f33597b50 ip 0000739f33597b50 sp 000078a46655d908 error 15
Apr 3 23:25:31 ns1 kernel: php-cgi[22790]: segfault at 669ef0f71b50 ip 0000669ef0f71b50 sp 0000745f1a64d868 error 15
Apr 3 23:35:32 ns1 kernel: php-cgi[28640]: segfault at 6aed000beb50 ip 00006aed000beb50 sp 000072fee7288898 error 15
Apr 3 23:40:50 ns1 kernel: php-cgi[31515]: segfault at 687d52eddb50 ip 0000687d52eddb50 sp 000076b3554880f8 error 15
Apr 3 23:50:31 ns1 kernel: php-cgi[4825]: segfault at 6d60c5fd3b50 ip 00006d60c5fd3b50 sp 000077d99b19b9a8 error 15
Apr 3 23:55:49 ns1 kernel: php-cgi[7992]: segfault at 785e365e6b50 ip 0000785e365e6b50 sp 0000796b94f2fb68 error 15
Apr 4 01:13:57 ns1 kernel: php-cgi[20818]: segfault at 2a20880 ip 0000000002a20880 sp 0000719ed9ca49c8 error 15

 

[IgorG]

I would suggest you try to find what exact script crashes, and re-view the code in order to understand what modules, extensions are used there. And then disable/remove all extensions from this version of PHP (find lines in the php.ini beginning with extension=xxxx.so and comment them). After that check if the error disappears. If so, then one by one enable extension back and see if the error comes again. If it does not help, then you might need to re-read PHP handlers, change PHP versions, etc.
I think that you might need to try to disable something like ionCube, Zend modules, APC, memcached, etc. if you have it.
Maybe there is problem in specific php code of your scripts.
So, deep investigation is required. You can ask Plesk Support Team assistance.

 

[nisamudeen97]

Hi,

Thanks for the advise. I have done a detailed investigation with server websites as there is more than 100 websites I am unable to find the "exact script that is crashing php". Is there any specific steps to follow and find the root crashing script ?

Checked the server apache error log meanwhile I am unable to find any crash or max_execution_time exceeded information in server log in /var/log/httpd/ logs.

Look to hear back your advise for the same. In the meantime I will try disabling extensions one by one.

 

[AYamshanov]

Hi nisamudeen97,

Your post will more informative if you provide additional information about the server: `plesk version`, version of php/apache.

A helpful link about searching problem script: PHP :: Generating a gdb backtrace. Have you got corefiles from php-cgi?

 

[nisamudeen97]

OS ‪CentOS 6.8 (Final)‬
Plesk version 12.5.30
Server version: Apache/2.2.15 (Unix)

 

[Bitpalast]

I think that you might need to try to disable something like ionCube, Zend modules, APC, memcached, etc. if you have it.

I second this. If it is not caused by a hardware failure, it is very likely caused by a PHP module like ionCube or another similar configuration.

 

[IgorG]

caused by a hardware failure

BTW, scanning RAM with with memcheck, for example, would be good idea.

 

[nisamudeen97]

Hi,

Thanks for the advise. I have changed php version of all websites. Now we are monitoring the performance.

BTW, scanning RAM with with memcheck, for example, would be good idea.

we have already done memory testing using "memtester" and it is found good. Below is the test result. Test result is found to be the same in all the 5 checks.


memtester 1024 5
memtester version 4.2.1 (64-bit)
Copyright (C) 2010 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 1024MB (1073741824 bytes)
got 1024MB (1073741824 bytes), trying mlock ...locked.
Loop 1/5:
Stuck Address : ok
Random Value : ok
Compare XOR : ok
Compare SUB : ok
Compare MUL : ok
Compare DIV : ok
Compare OR : ok
Compare AND : ok
Sequential Increment: ok
Solid Bits : ok
Block Sequential : ok
Checkerboard : ok
Bit Spread : ok
Bit Flip : ok
Walking Ones : ok
Walking Zeroes : ok
8-bit Writes : ok
16-bit Writes : ok

 

[Bitpalast]

I have another question: Is this thread mainly for the segfault issue or are you more concerned about speed? In case of the later I suggest to analyse the PHP scripts of the site with microtime-measurements step-by-step, because only rarely that is a server issue. Today's systems are so fast that they resemble a whole data center thirty years ago. The main problem is that many programmers don't care about optimized code, so they are making many mistakes.

For example, frequently programmers are constructing loops in a very bad way, so that the software spends a lot of time in them. For instance SQL statements are often placed inside a loop, so that instead of a single SQL request with an appropriate WHERE x IN ('...','...','...') statement, each select is executed and processed separately. Or instead of using array_walk(), arrays are updated by for...each loops. Or multibyte string functions are used where ordinary string functions would also do (and save a lot of CPU time), or another thing that is frequently done is that variables are initialized inside a loop, but they always carry the same value, so what good does it do to repeat initialization each cycle? A lot of wasted resources. Such problems can be identified by measuring the execution time like

// at the start of a sequence
$t1 = microtime(true);
...
// PHP code to measure here
...
// at the end
$t2 = microtime(true);
error_log(($t2-$t1)*1000);

This is a very effective way to optimize scripts, because it shows you exactly where your script is spending a lot of time. Go top down, meaning start with a large bracket, then narrow down where it spends much time.

 

[nisamudeen97]

Is this thread mainly for the segfault issue

Yes, 100% concentrated on segfault

 

[nisamudeen97]

Hi,

I have disabled all ionCube, Zend modules and cache modules in all php versions in the server. I am now monitoring how it goes.
I will keep posted here.

 

[nisamudeen97]

Hi,

I have disabled all php modules. Still there is segfault error. How I can I find if it is being caused by a culprit php script and which one ?


Apr 9 04:58:15 ns1 kernel: php-cgi[1660]: segfault at 4d90070 ip 0000000004d90070 sp 0000735a13c120b8 error 15
Apr 9 09:08:39 ns1 kernel: php-cgi[21449]: segfault at 4d7b600 ip 0000000004d7b600 sp 000073ddc2c2dd98 error 15
Apr 9 12:20:48 ns1 kernel: php-cgi[16312]: segfault at 32509f0 ip 00000000032509f0 sp 000076330de62c98 error 15
Apr 9 12:24:10 ns1 kernel: php-cgi[17072]: segfault at 57cb250 ip 00000000057cb250 sp 000073b43de52458 error 15
Apr 9 12:40:31 ns1 kernel: php-cgi[25844]: segfault at 546d680 ip 000000000546d680 sp 0000761cd0e62fa8 error 15
Apr 9 12:45:14 ns1 kernel: php-cgi[28055]: segfault at 457ef30 ip 000000000457ef30 sp 00007ea1697ee668 error 15
Apr 9 12:55:40 ns1 kernel: php-cgi[1141]: segfault at 3181a30 ip 0000000003181a30 sp 00007dca82dd4d98 error 15
Apr 9 13:00:15 ns1 kernel: php-cgi[3849]: segfault at 3d1d4e0 ip 0000000003d1d4e0 sp 00007149d8abc4b8 error 15
Apr 9 18:43:50 ns1 kernel: php-cgi[13314]: segfault at 55eabf8 ip 00000000055eabf8 sp 000072049ad902f8 error 15
Apr 9 22:25:52 ns1 kernel: php-cgi[13727]: segfault at 5044d70 ip 0000000005044d70 sp 000077bfa5215a38 error 15
Apr 10 05:59:09 ns1 kernel: php-cgi[23486]: segfault at 5d67120 ip 0000000005d67120 sp 00007c1a1b160368 error 15
Apr 10 06:04:14 ns1 kernel: php-cgi[26005]: segfault at 34de150 ip 00000000034de150 sp 00007ad097122a08 error 15
Apr 10 08:23:56 ns1 kernel: php-cgi[25461]: segfault at 3183590 ip 0000000003183590 sp 00007618c8a6af98 error 15
Apr 10 10:39:24 ns1 kernel: php-cgi[23141]: segfault at 65edd80 ip 00000000065edd80 sp 00007981eb33a148 error 15
Apr 10 12:15:49 ns1 kernel: php-cgi[5985]: segfault at 5c40f70 ip 0000000005c40f70 sp 00007996c0e53be8 error 15
Apr 10 12:20:13 ns1 kernel: php-cgi[8013]: segfault at 33acec0 ip 00000000033acec0 sp 000076995c7db188 error 15
Apr 10 12:35:48 ns1 kernel: php-cgi[14798]: segfault at 4b393b0 ip 0000000004b393b0 sp 00007f329f32b1f8 error 15
Apr 10 13:29:10 ns1 kernel: php-cgi[8251]: segfault at 4073168 ip 0000000004073168 sp 00007e4a60d11fb8 error 15

 

[agaitan]

got same issue