502 Bad Gateway - nginx

[Nickk]

Bonjour,

*UPDATED : SOLVED*
J'ai cette erreur quand je tente de me connecter à mon serveur depuisla semaine dernière . Mes sites sur le dédié ne fonctionnant pas, j'ai bien essayé d'aller sur Plesk pour voir pourquoi, mais impossible d'y accéder. Pareil en FTP. J'ai pu accéder seulement via le SSH.

Lorsque je tente d'accéder à un site du serveur, j'ai le message "Welcome to nginx!". Étrange,surtout que je n'ai fait aucune intervention sur le serveur. J'ai mis à jour mon Plesk il y a 5 semaines suite à une modification de Plesk. Depuis ça fonctionner bien, jusqu'à la semaine dernière, maintenant tous les sites sont inaccessibles !!!!

Quelqu'un peut m'aider sur cette erreur 502 Bad Gateway - nginx. ?
C'est pas dû à une nouvelle modification des caches ou un truc du genre ? Merci Plesk

 

[IgorG]

Sorry, but default language of this forum is English. Therefore please further write in English here. Thanks.

 

[Nickk]

Oupps sorry !
I've a 502 bad gateway issue since last week that block all websites on the host. I can't access Plesk interface, and FTP for the websites are not working anymore.
I only can access in SSH.
When I try to visit one of my website on the server, I've "Welcome to nginx!" error.

I haven't work on the server recently, so it doesn't come from a bad command or something. I think it's comming from an internal upgrade of Plesk.

Linux Plesk 11.0.9 Ubuntu 12.04 110120608.16

Anybody can help me on this, please ?

 

[IgorG]

Have you tried fix it with

# /usr/local/psa/admin/sbin/httpdmng --reconfigure-all

?

 

[Nickk]

I've tried 'do-release-upgrade'.
Not sure if it is the same thing at the end.

Don't know how to get this fix.

 

[Nickk]

Command '# /usr/local/psa/admin/sbin/httpdmng --reconfigure-all' in SSH show:

[2015-08-07 12:22:12] ERR [panel] SQLSTATE[HY000] [2002] No such file or directo  ry:
0: Abstract.php:144
  Zend_Db_Adapter_Pdo_Abstract->_connect()
1: Mysql.php:109
  Zend_Db_Adapter_Pdo_Mysql->_connect()
2: Abstract.php:459
  Zend_Db_Adapter_Abstract->query(string 'SET sql_mode = ''', array)
3: Abstract.php:238
  Zend_Db_Adapter_Pdo_Abstract->query(string 'SET sql_mode = ''', array)
4: Mysql.php:19
  Db_Adapter_Pdo_Mysql->query(string 'SET sql_mode = ''')
5: Abstract.php:93
  CommonPanel_Application_Abstract::initDbAdapter()
6: Helper.php:150
  Plesk\Session\Helper::initStorage()
7: Abstract.php:36
  CommonPanel_Application_Abstract->run()
8: Abstract.php:18
  CommonPanel_Application_Abstract::init()
9: AbstractCu.php:1479
  AbstractCu::initCLI()
10: httpdmng:9
ERROR: Zend_Db_Adapter_Exception: SQLSTATE[HY000] [2002] No such file or directo  ry (Abstract.php:144)

 

[IgorG]

Check that mysql server is up and running and you can connect.

 

[Nickk]

Thanks Igor for the advice.

I launch '# /etc/init.d/mysqld status' and return :

-bash: /etc/init.d/mysqld: No such file or directory

And...'# /etc/init.d/mysqld start' is returning the same issue.

Command '# service mysqld start' return :

mysqld: unrecognized service

Command '# service mysql start' and '# start mysql' both return :

start: Job failed to start

 

[Nickk]

In auth.log, I've for the first time an error from connecting databases on july 28th at 23h33 :

Jul 28 23:33:51 sshd[30611]: Unable to connect to Plesk Database: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)

Which does correspond quite well to the issue "ERROR: Zend_Db_Adapter_Exception: SQLSTATE[HY000] [2002] No such file or directory (Abstract.php:144)".

In systlog, at the same time , I've a 'Call Trace' which seems to indicate a kernel crash.

Jul 28 23:33:50 ks200730 kernel: Call Trace:
...
Jul 28 23:33:50 ks200730 kernel: Out of memory: Kill process 16638 (mysqld) score 97 or sacrifice child
Jul 28 23:33:50 ks200730 kernel: Killed process 16638 (mysqld) total-vm:1344640kB, anon-rss:161036kB, file-rss:0kB

I've Linux Plesk 11.0.9 Ubuntu 12.04 110120608.16

 

[UFHH01]

Hi Nickk,

your system ran out of memory and it decided, that killing "mysqld" will free the most of some used memory. You will either have to investgate the memory issue, or you could try a reboot, to see, if the damon will still be killed afterwards again.

 

[Nickk]

Thank you for your help and your inputs.
As metionned by you, I kill "mysqld" with comand '# sudo /etc/init.d/mysql stop' and try an other reboot.
But unfortunately, it isn't enough.
Do you think removing some datas from /var/www/vhosts will be a good try ?

I'm a little bite worry about not finding mysqld.sock anywhere on the server. I run a command '# find / -name mysqld' and get :

/usr/sbin/mysqld
/run/mysqld

But nothing in folder /run/mysqld and a file mysqld in /usr/sbin/ but not the mysqld.sock.
Does it seems normal for you ?

 

[UFHH01]

Hi Nickk,

you always could check running services with the example command:

ps -aux | grep mysql​

Another method to check the actual status for a service is:

service mysql status​

With the output, you can now check, if the process "mysqld" is actually running and if this is the case, there should be a corresponding "*.sock" or ".pid" file ( depending on your configuration settings for the service - you could have a look inside your configuration and check, which setting you defined there for the location and creation. If you don't find the corresponding pid and/or sock - files, it normally is a very good indication, that the service is not running at all. So please check, WHY the process got killed and not able to restart.

You don't necessarily have to delete data on your disk, if the disk has got a usage < 90 %. Please check you disk usage with:

df -hl
​

... and only delete data in temp - folders first, before starting to clean up useless stuff.

.... BUT ^^ ... I think you might have misunderstood the "memory" word. In this case, there is maybe missing RAM or misconfigured system configurations and/or settings, which lead to the killing process as described. Please have a look at a nice Oracle tutorial for the Out-of-memory-killer in Linux:

How to Configure the Linux Out-of-Memory Killer ( Online tutorial from Robert Chase )​

 

[Nickk]

Thanks UFHH01,

Command '# ps -aux | grep mysql' does :

root  7426  0.0  0.0  4436  736 ?  S  21:52  0:00 /bin/sh /usr/bin/mysqld_safe --innodb_force_recovery 4

mysql  7882  0.1  4.5 550016 92820 ?  Sl  21:52  0:07 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --innodb-force-recovery 4 --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306

root  17988  0.0  0.0  10464  924 pts/2  R+  23:03  0:00 grep --color=auto mysql

Command '# service mysql status' show:

Usage: /etc/init.d/mysql start|stop|restart|reload|force-reload|status

And '# service mysql status' :

 * /usr/bin/mysqladmin  Ver 8.42 Distrib 5.5.44, for debian-linux-gnu on x86_64
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Server version  5.5.44-0ubuntu0.14.04.1
Protocol version  10
Connection  Localhost via UNIX socket
UNIX socket  /var/run/mysqld/mysqld.sock
Uptime:  1 hour 2 min 28 sec

Threads: 1  Questions: 12268  Slow queries: 0  Opens: 2055  Flush tables: 2  Open tables: 2  Queries per second avg: 3.273

It's running !

Command '# df -hl' show:

Filesystem Size   Used Avail Use%  Mounted on
/dev/root  20G   2,1G  17G   12%   /
devtmpfs   993M  4,0K  993M  1%    /dev
none       4,0K  0     4,0K  0%    /sys/fs/cgroup
none       199M  1,1M  198M  1%    /run
none       5,0M  0     5,0M  0%    /run/lock
none       994M  8,0K  994M  1%    /run/shm
none       100M  0     100M  0%    /run/user
/dev/sda2  897G  34G   818G  4%    /var

The statut information from file located /var/log/mysql/error.log :

Status information:

Current dir: /var/lib/mysql/
Running threads: 0  Stack size: 196608

Key caches:
default
Buffer_size:  16777216
Block_size:  1024
Division_limit:  100
Age_limit:  300
blocks used:  3
not flushed:  0
w_requests:  0
writes:  0
r_requests:  6
reads:  3

handler status:
read_key:  1
read_next:  1
read_rnd  0
read_first:  3
write:  0
delete  0
update:  0

Table status:
Opened tables:  33
Open tables:  26
Open files:  18
Open streams:  0

Alarm status:
Active alarms:  0
Max used alarms: 0
Next alarm time: 0

Memory status:
Non-mmapped space allocated from system: 10579968
Number of free chunks:        20
Number of fastbin blocks:      20
Number of mmapped regions:      17
Space in mmapped regions:      49676288
Maximum total allocated space:      0
Space available in freed fastbin blocks: 1424
Total allocated space:        8531120
Total free space:        2048848
Top-most, releasable space:      1767808
Estimated memory (with thread stack):  60256256

Events status:
LLA = Last Locked At  LUA = Last Unlocked At
WOC = Waiting On Condition  DL = Data Locked

Event scheduler status:
State  : INITIALIZED
Thread id  : 0
LLA  : n/a:0
LUA  : n/a:0
WOC  : NO
Workers  : 0
Executed  : 0
Data locked: NO

Event queue status:
Element count  : 0
Data locked  : NO
Attempting lock : NO
LLA  : init_queue:139
LUA  : init_queue:149
WOC  : NO
Next activation : never
150807 21:56:21 mysqld_safe A mysqld process already exists
InnoDB: A new raw disk partition was initialized or
InnoDB: innodb_force_recovery is on: we do not allow
InnoDB: database modifications by the user. Shut down
InnoDB: mysqld and edit my.cnf so that newraw is replaced
InnoDB: with raw, and innodb_force_... is removed.
InnoDB: A new raw disk partition was initialized or
InnoDB: innodb_force_recovery is on: we do not allow
InnoDB: database modifications by the user. Shut down
InnoDB: mysqld and edit my.cnf so that newraw is replaced
InnoDB: with raw, and innodb_force_... is removed.
InnoDB: A new raw disk partition was initialized or
InnoDB: innodb_force_recovery is on: we do not allow
InnoDB: database modifications by the user. Shut down
InnoDB: mysqld and edit my.cnf so that newraw is replaced
InnoDB: with raw, and innodb_force_... is removed.
150807 21:58:49  InnoDB: Error: table `horde`.`#sql-7d88_72` does not exist in the InnoDB internal
InnoDB: data dictionary though MySQL is trying to drop it.
InnoDB: Have you copied the .frm file of the table to the
InnoDB: MySQL database directory from another database?
InnoDB: You can look for further help from

From the Oracle How to Configure the Linux Out-of-Memory Killer page (link) :
Command '# free -lm' show :

            total  used  free  shared  buffers  cached
Mem:        1986  1490  495  12  491  517
Low:        1986  1490  495
High:        0  0  0
-/+ buffers/cache:    481  1504
Swap:       512  0  512

# vmstat -SM 45 10

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r   b  swpd  free  buff  cache  si  so  bi  bo  in     cs   us sy id wa st
0  0  0       493  493   517     0  0  21  16   110  151  1  0  98  1  0
0  0  0       491  493   517     0  0  0    6   202  251  1  0  98  0  0
0  0  0       492  493   517     0  0  0    7   177  212  1  0  98  0  0
0  0  0       491  493   517     0  0  0    7   173  207  1  0  98  0  0
0  0  0       488  493   517     0  0  0    8   218  267  2  1  98  0  0
0  0  0       482  493   517     0  0  0    6   225  274  2  0  98  0  0
0  0  0       491  493   517     0  0  0    8   178  214  1  0  98  0  0
1  0  0       489  493   518     0  0  0    8   200  242  1  0  98  0  0
0  0  0       488  494   518     0  0  0    9   183  224  1  0  98  0  0
0  0  0       490  494   518     0  0  0    6   190  229  1  0  98  0  0

Memory
swpd: the amount of virtual memory used.
free: the amount of idle memory.
buff: the amount of memory used as buffers.
cache: the amount of memory used as cache.
inact: the amount of inactive memory. (-a option)
active: the amount of active memory. (-a option)

Swap
si: Amount of memory swapped in from disk (/s).
so: Amount of memory swapped to disk (/s).

 

[UFHH01]

Hi Nickk,

maybe this tutorial helps you to solve your MySQL issue:

mysql failed to start due to corrupted innoDB ( external link, please inform me, when the link is going dead )
​

For further investigations and solutions with a corrupted MySQL - database, please consider opening a support ticket with Odin, because this really needs some deep investigations on your server and may take a while. ( Odin / Parallels - Support )


Only "495" free RAM is not really good for a production hosting server. Consider upgrading your contract.

 

[Nickk]

Thanks for your support.
I open a ticket with Plesk/Odin.

For the memory, I'll upgrading my contract or remove some wesites, as soon as I get my server back.
Thank you once again for your help.

 

[Nickk]

Issue solved by Plesk support.
Plesk was incomplete on the server after executing 'apt-get dist-upgrade'.
See Plesk 12 Release Notes and this post.