ERROR: An another FPM instance seems to already listen on

[zwt]

Hello,

We are keep having a weird issue with PHP-FPM for some time now. After reboots and from time to time PHP-FPM restarts fails and causing downtime for some of the sites. Looking at the logs we see the followings:

2017/08/01 08:36:59 [crit] 5647#0: *14572 connect() to unix:///var/www/vhosts/system/rauchringking.de/php-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 78.139.25.250, server: rauchringking.de, request: "GET /shisha-kaminkopf HTTP/1.1", upstream: "fastcgi://unix:///var/www/vhosts/system/rauchringking.de/php-fpm.sock:", host: "www.rauchringking.de", referrer: "Finde den passendes Shisha Kaminkopf - Praxistipps zur Verwendung!"
root@webserver plesk-php70-fpm]# tail -f error.log
[01-Aug-2017 05:59:13] WARNING: [pool rauchringking.de] child 2302 said into stderr: " AND "
[01-Aug-2017 05:59:13] WARNING: [pool rauchringking.de] child 2302 said into stderr: " ( mt3.meta_ke..."
[01-Aug-2017 05:59:15] WARNING: [pool ledschuhe.net] child 2340 said into stderr: "ERROR: Unable to set php_value 'apc.enabled'"
[01-Aug-2017 05:59:50] WARNING: [pool dein-vibrator.de] child 2403 said into stderr: "ERROR: Unable to set php_value 'apc.enabled'"
[01-Aug-2017 06:00:58] WARNING: [pool dein-vibrator.de] child 2555 said into stderr: "ERROR: Unable to set php_value 'apc.enabled'"
[01-Aug-2017 06:01:16] WARNING: [pool ledschuhe.net] child 2604 said into stderr: "ERROR: Unable to set php_value 'apc.enabled'"
[01-Aug-2017 06:02:14] NOTICE: Terminating ...
[01-Aug-2017 06:02:14] NOTICE: exiting, bye-bye!
[01-Aug-2017 06:09:34] ERROR: An another FPM instance seems to already listen on /var/www/vhosts/system/blumenversand-international.de/php-fpm.sock
[01-Aug-2017 06:09:34] ERROR: FPM initialization failed
root@webserver plesk-php70-fpm]# service plesk-php70-fpm restart
Stopping php-fpm: [FAILED]
Starting php-fpm: [01-Aug-2017 08:40:20] ERROR: An another FPM instance seems to already listen on /var/www/vhosts/system/blumenversand-international.de/php-fpm.sock
[01-Aug-2017 08:40:20] ERROR: FPM initialization failed
[FAILED]
[root@webserver plesk-php70-fpm]# rm -rf /var/www/vhosts/system/blumenversand-international.de/php-fpm.sock
[root@webserver plesk-php70-fpm]# service plesk-php70-fpm restart
Stopping php-fpm: [FAILED]
Starting php-fpm: [01-Aug-2017 08:40:30] ERROR: An another FPM instance seems to already listen on /var/www/vhosts/system/laufradkids.de/php-fpm.sock
[01-Aug-2017 08:40:30] ERROR: FPM initialization failed
[FAILED]
[root@webserver plesk-php70-fpm]# rm -rf /var/www/vhosts/system/laufradkids.de/php-fpm.sock
[root@webserver plesk-php70-fpm]# service plesk-php70-fpm restart
Stopping php-fpm: [FAILED]
Starting php-fpm: [01-Aug-2017 08:40:37] ERROR: An another FPM instance seems to already listen on /var/www/vhosts/system/ledschuhe.net/php-fpm.sock
[01-Aug-2017 08:40:37] ERROR: FPM initialization failed
[FAILED]
[root@webserver plesk-php70-fpm]# rm -rf /var/www/vhosts/system/ledschuhe.net/php-fpm.sock
[root@webserver plesk-php70-fpm]# service plesk-php70-fpm restart
Stopping php-fpm: [FAILED]
Starting php-fpm: [01-Aug-2017 08:40:44] ERROR: An another FPM instance seems to already listen on /var/www/vhosts/system/preiswertkaufen.org/php-fpm.sock
[01-Aug-2017 08:40:44] ERROR: FPM initialization failed
[FAILED]
[root@webserver plesk-php70-fpm]# rm -rf /var/www/vhosts/system/preiswertkaufen.org/php-fpm.sock
[root@webserver plesk-php70-fpm]# service plesk-php70-fpm restart
Stopping php-fpm: [FAILED]
Starting php-fpm: [01-Aug-2017 08:40:53] ERROR: An another FPM instance seems to already listen on /var/www/vhosts/system/vlog.xn--online-rzteberatung-mwb.de/php-fpm.sock
[01-Aug-2017 08:40:53] ERROR: FPM initialization failed
[FAILED]
[root@webserver plesk-php70-fpm]# rm -rf /var/www/vhosts/system/vlog.xn--online-rzteberatung-mwb.de/php-fpm.sock
[root@webserver plesk-php70-fpm]# service plesk-php70-fpm restart
Stopping php-fpm: [FAILED]
Starting php-fpm: [01-Aug-2017 08:41:00] ERROR: An another FPM instance seems to already listen on /var/www/vhosts/system/wasserpfeife-kaufen.net/php-fpm.sock
[01-Aug-2017 08:41:00] ERROR: FPM initialization failed
[FAILED]
[root@webserver plesk-php70-fpm]# rm -rf /var/www/vhosts/system/wasserpfeife-kaufen.net/php-fpm.sock
[root@webserver plesk-php70-fpm]# service plesk-php70-fpm restart
Stopping php-fpm: [FAILED]
Starting php-fpm: [01-Aug-2017 08:41:07] ERROR: An another FPM instance seems to already listen on /var/www/vhosts/system/xn--online-rzteberatung-mwb.de/php-fpm.sock
[01-Aug-2017 08:41:07] ERROR: FPM initialization failed
[FAILED]
[root@webserver plesk-php70-fpm]# rm -rf /var/www/vhosts/system/xn--online-rzteberatung-mwb.de/php-fpm.sock
[root@webserver plesk-php70-fpm]# service plesk-php70-fpm restart
Stopping php-fpm: [FAILED]
Starting php-fpm: [ OK ]
[root@webserver plesk-php70-fpm]# service plesk-php70-fpm restart
Stopping php-fpm: [ OK ]
Starting php-fpm: [ OK ]
[root@webserver plesk-php70-fpm]#


We can only restart PHP-FPM if we kill the socket files. Every time this is happening with different sites so it's totally random. Can you advise?

OS is CentOS 6.5

 

[zwt]

We just noticed that if we restart PHP7 this way we lose Additional nginx directives set earlier for domains using PHP7 and this is a pretty big issues. Websites stops working due to this.

 

[zwt]

nobody?

 

[Bitpalast]

Why are there so many consecutive PHP-FPM restarts in your error log? This is unusual. Normally, if a restart is due, it should only be executed once. The reason for the error is that one process (loading or unloading) is overlapping with the next. The root cause is that something is causing several such reloads. Could this be caused by an extension your system is using or an event that is reacting on a certain condition?

 

[zwt]

The reloads i.e PHP-FPM restarts are manual caused by myself and you can see the reason why. The PHP-FPM sockets are preventing the restarts as long as I don't manually remove them which can be seen the in my paste. After removing a few of them PHP-FPM is starting but every time this is random how many and which sockets we need to remove to start this again.

 

[Bitpalast]

What happens when you try
# service plesk-php<version>-fpm stop
then
# service plesk-php<version>-fpm start
as separate commands? To me it does not make sense that PHP-FPM keeps listening to sockets when the process has ended. If the process is not there, it cannot listen on a socket. The only explanation for it is that there are more than one master processes active. Are you using multiple PHP-FPM master processes, e.g. as described in A better way to run PHP-FPM ?

 

[zwt]

We have the same issue if we do this with 2 commands as well. We are using the default PHP-FPM setup what Plesk provides, no customisation. As I remember we saw some segfaults in dmesg as well which could be the culprit here however we need to find out how we can replicate this.

 

[Bitpalast]

It is possible that the service crashes. Please try to increase the system semaphores as described in section (3) of Input - How to overcome intermittent, occasional 502 errors due to Apache or PHP-FPM response failures, then restart PHP-FPM and wait if it happens again.

 

[zwt]

Thanks Peter. We are applying this and closely monitor PHP-FPM processes.

 

[zwt]

Peter,

We just rebooted the server to replicate this and unfortunately even after your suggestions this is still happening:

[root@webserver init.d]# service php-fpm status
php-fpm (pid 2665) is running...
[root@webserver init.d]# service plesk-php56-fpm status
php-fpm (pid 2676) is running...
[root@webserver init.d]# service plesk-php70-fpm status
php-fpm dead but subsys locked
[root@webserver init.d]# service plesk-php7-fpm restart
plesk-php7-fpm: unrecognized service
[root@webserver init.d]# service plesk-php70-fpm restart
Stopping php-fpm: [FAILED]
Starting php-fpm: [06-Aug-2017 08:54:53] ERROR: An another FPM instance seems to already listen on /var/www/vhosts/system/blumenversand-international.de/php-fpm.sock
[06-Aug-2017 08:54:53] ERROR: FPM initialization failed
[FAILED]
[root@webserver init.d]# ps aux | grep plesk-php70-fpm
root 4192 0.0 0.0 105364 904 pts/0 S+ 08:55 0:00 grep plesk-php70-fpm
[root@webserver init.d]# ps aux | grep php-fpm
root 2665 0.0 0.0 415576 6656 ? Ss 08:45 0:00 php-fpm: master process (/etc/php-fpm.conf)
root 2676 0.0 0.0 486936 9336 ? Ss 08:45 0:00 php-fpm: master process (/opt/plesk/php/5.6/etc/php-fpm.conf)
10244 4190 7.1 0.1 544708 74544 ? S 08:55 0:00 php-fpm: pool preiswertkaufen.org
10244 4193 7.8 0.0 511676 51080 ? S 08:55 0:00 php-fpm: pool preiswertkaufen.org
root 4196 0.0 0.0 105368 904 pts/0 S+ 08:55 0:00 grep php-fpm
[root@webserver init.d]# /var/www/vhosts/system/blumenversand-international.de/php-fpm.sock
-bash: /var/www/vhosts/system/blumenversand-international.de/php-fpm.sock: Permission denied
[root@webserver init.d]# ls -l /var/www/vhosts/system/blumenversand-international.de/php-fpm.sock
srw-rw---- 1 root psaserv 0 Aug 6 08:45 /var/www/vhosts/system/blumenversand-international.de/php-fpm.sock
[root@webserver init.d]#

Can you advise further?

Regards,
Andrew

 

[zwt]

We tried to stop and start with 2 commands but no luck:

[root@webserver init.d]# service plesk-php70-fpm stop
Stopping php-fpm: [FAILED]
[root@webserver init.d]# service plesk-php70-fpm start
Starting php-fpm: [06-Aug-2017 08:58:31] ERROR: An another FPM instance seems to already listen on /var/www/vhosts/system/blumenversand-international.de/php-fpm.sock
[06-Aug-2017 08:58:31] ERROR: FPM initialization failed
[FAILED]
[root@webserver init.d]#

 

[zwt]

The process list is also attached.

 

[zwt]

Additional information that if I stop plesk-php56-fpm after that I'm able to start plesk-php70-fpm without any problem however 5.6 fails to start with the same error thereafter:

root@webserver log]# service php-fpm stop
Stopping php-fpm: [ OK ]
[root@webserver log]# service plesk-php70-fpm start
Starting php-fpm: [06-Aug-2017 09:02:42] ERROR: An another FPM instance seems to already listen on /var/www/vhosts/system/blumenversand-international.de/php-fpm.sock
[06-Aug-2017 09:02:42] ERROR: FPM initialization failed
[FAILED]
[root@webserver log]# service plesk-php56-fpm stop
Stopping php-fpm: [ OK ]
[root@webserver log]# service plesk-php70-fpm start
Starting php-fpm: [ OK ]
[root@webserver log]# service plesk-php56-fpm status
php-fpm is stopped
[root@webserver log]# service plesk-php70-fpm status
php-fpm (pid 5146) is running...
[root@webserver log]# service php-fpm status
php-fpm is stopped
[root@webserver log]# service plesk-php56-fpm status
php-fpm is stopped
[root@webserver log]# service plesk-php56-fpm start
Starting php-fpm: [06-Aug-2017 09:04:10] ERROR: An another FPM instance seems to already listen on /var/www/vhosts/system/blumenversand-international.de/php-fpm.sock
[06-Aug-2017 09:04:10] ERROR: FPM initialization failed
[FAILED]
[root@webserver log]# service php-fpm start
Starting php-fpm: [06-Aug-2017 09:04:35] ERROR: An another FPM instance seems to already listen on /var/www/vhosts/system/ledschuhe.net/php-fpm.sock
[06-Aug-2017 09:04:35] ERROR: FPM initialization failed
[FAILED]
[root@webserver log]#

Both blumenversand-international.de and ledschuhe.net use PHP 7. Attached the logs how we could start php-fpm56 finally.

 

[Bitpalast]

What is happening is that the PHP-FPM service is not stopping correctly. Parts remain active, indicated by the symptom that "another FPM instance seems to already listen on" the socket and the status message "php-fpm dead but subsys locked".

Please look into
/opt/plesk/php/<version>/etc/php-fpm.d/<domain>.conf
for example
/opt/plesk/php/5.6/etc/php-fpm.d/blumenversand-international.de.conf
and check, if these parameters are set correctly:
listen = php-fpm.sock
listen.owner = root
listen.group = psaserv
listen.mode = 0660
pm = ondemand
If these are set to different values, try to manually change and restart the service.

Please check, if the PID file location is correct in /opt/plesk/php/<version>/etc/php-fpm.d/php-fpm.conf
pid = /run/plesk-php<version>-fpm.pid
Check if the .pid file exists when the service is active.
Can you display its content
# less /run/plesk-php<version>-fpm.pid
?

selinux could be a source of trouble, too. It might prevent the service from stopping, because it could block access to files that need to be accessed for it. If you are using it, disable it for testing.

 

[zwt]

Hi,


[root@webserver init.d]# service plesk-php70-fpm restart
Stopping php-fpm: [FAILED]
Starting php-fpm: [06-Aug-2017 11:19:55] ERROR: An another FPM instance seems to already listen on /var/www/vhosts/system/blumenversand-international.de/php-fpm.sock
[06-Aug-2017 11:19:55] ERROR: FPM initialization failed
[FAILED]
[root@webserver init.d]# cat /opt/plesk/php/5.6/etc/php-fpm.d/blumenversand-international.de.conf
; ATTENTION!
;
; DO NOT MODIFY THIS FILE BECAUSE IT WAS GENERATED AUTOMATICALLY,
; SO ALL YOUR CHANGES WILL BE LOST THE NEXT TIME THE FILE IS GENERATED.

; If you need to customize this file, use either custom PHP settings tab in
; Panel or override settings in /var/www/vhosts/system/blumenversand-international.de/conf/php.ini.
; To override pool configuration options, specify them in [php-fpm-pool-settings]
; section of /var/www/vhosts/system/blumenversand-international.de/conf/php.ini file.

[blumenversand-international.de]

; Don't override following options, they are relied upon by Plesk internally

prefix = /var/www/vhosts/system/$pool
user = blumenversand
group = psacln

listen = php-fpm.sock
listen.owner = root
listen.group = psaserv
listen.mode = 0660

; Following options can be overridden

chdir = /

; By default use ondemand spawning (this requires php-fpm >= 5.3.9)

pm = ondemand
pm.max_children = 5
pm.process_idle_timeout = 10s

; Following pm.* options are used only when 'pm = dynamic'

pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 1

; Uses for log facility
; If php_value[error_log] is not defined error output will be send for nginx
catch_workers_output = yes
; php.ini custom configuration directives
php_value[error_reporting] = 22519
php_value[open_basedir] = "/var/www/vhosts/blumenversand-international.de/:/tmp/"
[root@webserver init.d]#

The config file of PHP 7.0 is identical compared to 5.6. The PID file locations for all PHP config files (5.6 and 7.0) seems to be correct.

[root@webserver etc]# ls -l /var/run/plesk-php56-fpm.pid
-rw-r--r-- 1 root root 4 Aug 6 11:18 /var/run/plesk-php56-fpm.pid
[root@webserver run]# service php-fpm status
php-fpm (pid 2638) is running...
[root@webserver run]# service plesk-56-fpm status
plesk-56-fpm: unrecognized service
[root@webserver run]# service plesk-php56-fpm status
php-fpm (pid 2650) is running...
[root@webserver run]# service plesk-php70-fpm status
php-fpm dead but subsys locked
[root@webserver run]#

Selinux is disabled on the server.

 

[Bitpalast]

You will need to find the reason why the PHP-FPM service cannot access files it needs to access when stopping. These will be at least, not limited to its own configuration files and the php-fpm.sock file plus the .pid file.

Are you seeing "too many open files" in any log, especially the Plesk php-fpm logs? Meaning, does the server have enough spare file descriptors? Do you have enough spare inodes? Compare "# lsof | wc -l" to number of open files allowed on the system "sysctl -p" or content of fs.file-max in /etc/sysctl.conf and possible limits set in /etc/security/limits.conf.

Are your websites located in the Plesk virtual host default path, /var/www/vhosts/...? Have any changes be made to the /var/www/vhosts/system path permissions? Is any other security software active on the server besides SElinux? Could it be possible that a second PHP installation is present, maybe disfunctional, but maybe in another path so that the server is not really using the Plesk PHP installation?

# /usr/local/psa/bin/php_handler --list
should yield something similar to

                  id:        display name:  full version:  version:    type:                         cgi-bin:                    php-cli:                        php.ini:  custom:  status:
                  cgi  5.4.16 by OS vendor         5.4.16       5.4      cgi                 /usr/bin/php-cgi                /usr/bin/php                    /etc/php.ini    false disabled
              fastcgi  5.4.16 by OS vendor         5.4.16       5.4  fastcgi                 /usr/bin/php-cgi                /usr/bin/php                    /etc/php.ini    false  enabled
                  fpm  5.4.16 by OS vendor         5.4.16       5.4      fpm                /usr/sbin/php-fpm                /usr/bin/php                    /etc/php.ini    false  enabled
               module  5.4.16 by OS vendor         5.4.16       5.4   module                 /usr/bin/php-cgi                /usr/bin/php                    /etc/php.ini    false disabled
      plesk-php56-cgi               5.6.31         5.6.31       5.6      cgi   /opt/plesk/php/5.6/bin/php-cgi  /opt/plesk/php/5.6/bin/php  /opt/plesk/php/5.6/etc/php.ini     true disabled
  plesk-php56-fastcgi               5.6.31         5.6.31       5.6  fastcgi   /opt/plesk/php/5.6/bin/php-cgi  /opt/plesk/php/5.6/bin/php  /opt/plesk/php/5.6/etc/php.ini     true  enabled
      plesk-php56-fpm               5.6.31         5.6.31       5.6      fpm  /opt/plesk/php/5.6/sbin/php-fpm  /opt/plesk/php/5.6/bin/php  /opt/plesk/php/5.6/etc/php.ini     true  enabled
      plesk-php70-cgi               7.0.21         7.0.21       7.0      cgi   /opt/plesk/php/7.0/bin/php-cgi  /opt/plesk/php/7.0/bin/php  /opt/plesk/php/7.0/etc/php.ini     true disabled
  plesk-php70-fastcgi               7.0.21         7.0.21       7.0  fastcgi   /opt/plesk/php/7.0/bin/php-cgi  /opt/plesk/php/7.0/bin/php  /opt/plesk/php/7.0/etc/php.ini     true  enabled
      plesk-php70-fpm               7.0.21         7.0.21       7.0      fpm  /opt/plesk/php/7.0/sbin/php-fpm  /opt/plesk/php/7.0/bin/php  /opt/plesk/php/7.0/etc/php.ini     true  enabled
      plesk-php71-cgi                7.1.7          7.1.7       7.1      cgi   /opt/plesk/php/7.1/bin/php-cgi  /opt/plesk/php/7.1/bin/php  /opt/plesk/php/7.1/etc/php.ini     true disabled
  plesk-php71-fastcgi                7.1.7          7.1.7       7.1  fastcgi   /opt/plesk/php/7.1/bin/php-cgi  /opt/plesk/php/7.1/bin/php  /opt/plesk/php/7.1/etc/php.ini     true  enabled
      plesk-php71-fpm                7.1.7          7.1.7       7.1      fpm  /opt/plesk/php/7.1/sbin/php-fpm  /opt/plesk/php/7.1/bin/php  /opt/plesk/php/7.1/etc/php.ini     true  enabled

without duplicates for the corresponding versions. If there are more handlers registered, maybe duplicates or false paths etc., the handlers should be corrected before continuing with the investigation.

You can also try this: When the service is in the "php-fpm dead but subsys locked" state, run
# ps aux | grep php/7.0
then note the PID of that process (there should only be one in the output) and do
# lsof -p <PID>
to list all files that the process is still using. This could give more clues on what is going on.

And I'd recommend to make sure you are on the lastest Plesk modules, too:
# plesk installer --select-release-current --reinstall-patch --upgrade-installed-components
If this throws an error, it is probably a system library that has a wrong version, a dependency that isn't right.

 

[zwt]

We don't see too many open files errors in the log files. Yes the server has enough inodes as well as space available. Yes the document root is in /var/www/vhosts and no customisation has been made nor in the config files neither permission wise. There are no firewall, grsec kernel or anything else installed on the server.

The output of PHP handlers has been attached and as far as we can see there are no duplicates for the corresponding versions. I have also attached the process list when PHP-FPM dead but subsys locked and we didn't see any related process either.

[root@webserver init.d]# ps aux | grep plesk-php70-fpm
root 4192 0.0 0.0 105364 904 pts/0 S+ 08:55 0:00 grep plesk-php70-fpm
[root@webserver init.d]# ps aux | grep php-fpm
root 2665 0.0 0.0 415576 6656 ? Ss 08:45 0:00 php-fpm: master process (/etc/php-fpm.conf)
root 2676 0.0 0.0 486936 9336 ? Ss 08:45 0:00 php-fpm: master process (/opt/plesk/php/5.6/etc/php-fpm.conf)
10244 4190 7.1 0.1 544708 74544 ? S 08:55 0:00 php-fpm: pool preiswertkaufen.org
10244 4193 7.8 0.0 511676 51080 ? S 08:55 0:00 php-fpm: pool preiswertkaufen.org
root 4196 0.0 0.0 105368 904 pts/0 S+ 08:55 0:00 grep php-fpm
[root@webserver init.d]#

We have ran "plesk installer --select-release-current --reinstall-patch --upgrade-installed-components" and we received no warnings or errors:

All patches were applied.
Patches were installed successfully.
The changes were applied successfully.

 

[Bitpalast]

It could be a misconfiguration of the FastCGI values. However, at this level of detail I recommend to open a support ticket with Plesk and let the support solve it.