Issue DB query failed: SQLSTATE[HY000] [2002] Connection refused: after Obsidian update 18.0.53

[Christopher-James]

Server operating system version

Ubuntu 22.04.2 LTS

Plesk version and microupdate number

Plesk Obsidian 18.0.53

Hi

Everyday since i updated to Obsidian mariadb keeps stopping with the error DB query failed: SQLSTATE[HY000] [2002] Connection refused:

Below is the error from the PLESK log file, i have no idea why.

Logging into plesk repair and just restarting mariadb brings it back but I know it will happen again.

DB query failed: SQLSTATE[HY000] [2002] Connection refused:
0: /opt/psa/admin/plib/Db/Adapter/Pdo/Mysql.php:79
    Db_Adapter_Pdo_Mysql->query(string 'SET sql_mode = ''')
1: /opt/psa/admin/plib/CommonPanel/Application/Abstract.php:113
    CommonPanel_Application_Abstract::initDbAdapter()
2: /opt/psa/admin/plib/Session/Helper.php:176
    Plesk\Session\Helper::initStorage()
3: /opt/psa/admin/plib/CommonPanel/Application/Abstract.php:48
    CommonPanel_Application_Abstract->run()
4: /opt/psa/admin/plib/CommonPanel/Application/Abstract.php:30
    CommonPanel_Application_Abstract::init()
5: /opt/psa/admin/plib/pm/Bootstrap.php:18
    pm_Bootstrap::init()
6: /opt/psa/admin/plib/sdk.php:11
    require_once(string '/opt/psa/admin/plib/sdk.php')
7: /opt/psa/admin/htdocs/modules/rest-api/public/index.php:4

 

[Christopher-James]

Repairing PLESK gives the error below with FAILED

Reconfiguring the Plesk installation Started bootstrapper repair procedure. This may take a while. Certain actions may be skipped if not applicable. Finishing up upgrade procedures and rerunning previously failed upgrade actions... ===> Cumulative Plesk database upgrade and repair (revertable stage) has been started. ===> Preparing Plesk database upgrade (revertable stage). ===> Cumulative upgrade and repair of Plesk database (revertable stage) has been completed. ===> Plesk database scheme upgrade has been started. Applying migrations from: /opt/psa/bootstrapper/pp18.0.53-bootstrapper/migrations/ ===> Plesk database scheme upgrade has been completed. ===> Cumulative Plesk upgrade and repair (final stage) has been started. ===> Preparing Plesk upgrade (final stage). ===> Cumulative upgrade and repair of Plesk (final stage) has been completed. Reconfiguring mail subsystem... Reconfiguring Apache web server... Reconfiguring ProFTPD FTP server... ===> Configuring ProFTPD server Reconfiguring AWStats web statistics... Reconfiguring WatchDog... Restoring SELinux contexts... Reconfiguring SSL ciphers and protocols... Regenerating web servers' configuration files... Cleaning active Panel sessions... Fixing permissions on Panel packages files ERROR:Unable to retrieve repository configuration from autoinstaller: autoinstaller is busy Bootstrapper repair finished. Errors occurred while performing the following actions: fix Plesk packages permissions. Check '/var/log/plesk/install/plesk_18.0.53_repair.log' and '/var/log/plesk/install/plesk_18.0.53_repair_problems.log' for details. If you can't resolve the issue on your own, please address Plesk support.

 

[Peter Debik]

"autoinstaller is busy" doesn't sound good. Maybe a previous installation process is hung? This can have many reasons, for example incomplete operating system updates. Do you see more information in /var/log/plesk/install/autoinstaller3.log?

Periodically stopping MariaDB can be caused by a corrupt table. In that case the database will work, but the moment that corrupt table is addressed, it crashes. You can find more information on that in /var/log/messages, e.g. by grep mariadb /var/log/messages. The table then needs to be repaired or removed and restored from a backup.

 

[Christopher-James]

"autoinstaller is busy" doesn't sound good. Maybe a previous installation process is hung? This can have many reasons, for example incomplete operating system updates. Do you see more information in /var/log/plesk/install/autoinstaller3.log?

Periodically stopping MariaDB can be caused by a corrupt table. In that case the database will work, but the moment that corrupt table is addressed, it crashes. You can find more information on that in /var/log/messages, e.g. by grep mariadb /var/log/messages. The table then needs to be repaired or removed and restored from a backup.

Hi Peter

So the autoinstaller3.log shows lots of warnings, a few as an example below, for some reason I cant attach the entire log file, I dont really know exactly what id be looking for though?

Warning: ez_template_suffix is NOT defined in  PHP80_17/php80-ubt22.04-x86_64.inf3

Warning: ez_template_suffix is NOT defined in  pool/PSA_18.0.53_12010/plesk-18.0.53-ubt22.04-x86_64.inf3

Warning: Failed to detach from parents cgroup, installer might be unexpectedly killed later by the system

Also when trying to access /var/log/messages i get "No such file or directory"

 

[Peter Debik]

Maybe you can start on the operating system level and check whether your operating system is up to date or whether there are installation errors, broken dependencies etc. Also for the MariaDB error, it is best to check /var/log/messages, because MariaDB is very stable. It rarely crashes, most of times it crashes for damaged tables.

 

[Christopher-James]

Hi @Peter Debik

So this is still an issue for me, the database crashed again today

Could not ping database connection: DB query failed: SQLSTATE[HY000] [2002] Connection refused

Then in the logs i have loads of errors

DB query failed: SQLSTATE[HY000] [2002] Connection refused:
0: /opt/psa/admin/plib/Db/Adapter/Pdo/Mysql.php:79
    Db_Adapter_Pdo_Mysql->query(string 'SET sql_mode = ''')
1: /opt/psa/admin/plib/CommonPanel/Application/Abstract.php:113
    CommonPanel_Application_Abstract::initDbAdapter()
2: /opt/psa/admin/plib/Db/Table/Broker.php:145
    Db_Table_Broker::_init()
3: /opt/psa/admin/plib/Db/Table/Broker.php:48
    Db_Table_Broker::get(string 'mailnames')
4: /opt/psa/admin/htdocs/mailconfig/index.php:125

    
DB query failed: SQLSTATE[HY000] [2002] Connection refused:
0: /opt/psa/admin/plib/Db/Adapter/Pdo/Mysql.php:79
    Db_Adapter_Pdo_Mysql->query(string 'SET sql_mode = ''')
1: /opt/psa/admin/plib/CommonPanel/Application/Abstract.php:113
    CommonPanel_Application_Abstract::initDbAdapter()
2: /opt/psa/admin/plib/Session/Helper.php:176
    Plesk\Session\Helper::initStorage()
3: /opt/psa/admin/plib/CommonPanel/Application/Abstract.php:48
    CommonPanel_Application_Abstract->run()
4: /opt/psa/admin/plib/CommonPanel/Application/Abstract.php:30
    CommonPanel_Application_Abstract::init()
5: /opt/psa/admin/plib/pm/Bootstrap.php:18
    pm_Bootstrap::init()
6: /opt/psa/admin/plib/sdk.php:11

DB query failed: SQLSTATE[HY000] [2002] Connection refused:
0: /opt/psa/admin/plib/Db/Adapter/Pdo/Mysql.php:79
    Db_Adapter_Pdo_Mysql->query(string 'SET sql_mode = ''')
1: /opt/psa/admin/plib/CommonPanel/Application/Abstract.php:113
    CommonPanel_Application_Abstract::initDbAdapter()
2: /opt/psa/admin/plib/Session/Helper.php:176
    Plesk\Session\Helper::initStorage()
3: /opt/psa/admin/plib/CommonPanel/Application/Abstract.php:48
    CommonPanel_Application_Abstract->run()
4: /opt/psa/admin/plib/CommonPanel/Application/Abstract.php:30
    CommonPanel_Application_Abstract::init()
5: /opt/psa/admin/plib/pm/Bootstrap.php:18
    pm_Bootstrap::init()
6: /opt/psa/admin/plib/sdk.php:11
    require_once(string '/opt/psa/admin/plib/sdk.php')
7: /opt/psa/admin/htdocs/modules/rest-api/public/index.php:4


    
DB query failed: SQLSTATE[HY000] [2002] Connection refused:
0: /opt/psa/admin/plib/Db/Adapter/Pdo/Mysql.php:79
    Db_Adapter_Pdo_Mysql->query(string 'SET sql_mode = ''')
1: /opt/psa/admin/plib/CommonPanel/Application/Abstract.php:113
    CommonPanel_Application_Abstract::initDbAdapter()
2: /opt/psa/admin/plib/Session/Helper.php:176
    Plesk\Session\Helper::initStorage()
3: /opt/psa/admin/plib/auth.php:275
    AutoPrepend->initUserSession()
4: /opt/psa/admin/plib/auth.php:188
    AutoPrepend->run()
5: /opt/psa/admin/plib/auth.php:583

Going through PLESK Repair i can restart the sql with no problems and everything comes back up

 

[Peter Debik]

You are looking at the panel.log file, but MariaDB messages logged to /var/log/messages (or /var/log/syslog). Please check there for the mariadb related entries.

 

[Christopher-James]

You are looking at the panel.log file, but MariaDB messages logged to /var/log/messages (or /var/log/syslog). Please check there for the mariadb related entries.

So this is the log from the time it went down to a few minutes after it was restarted, had to zip it as you cant upload txt or log files directly, to be honest I have no idea what im looking for

 

[Peter Debik]

The line you are looking for is
"A process of this unit has been killed by the OOM killer."
This points at a RAM issue. The next step here could be to examine, how mach free RAM you had on Jul 28 12:32:26. I guess it was completely used up by processes (not cache, but really used)?

 

[Christopher-James]

The line you are looking for is
"A process of this unit has been killed by the OOM killer."
This points at a RAM issue. The next step here could be to examine, how mach free RAM you had on Jul 28 12:32:26. I guess it was completely used up by processes (not cache, but really used)?

Hi @Peter Debik

Im not sure what to do with the info in the log. Ive looked at the OOMkiller info and found the process that was killed due to Out of memory but no idea what it points too?

Jul 28 12:32:26 relaxed-banach kernel: [6815598.018829] Tasks state (memory values in pages):
Jul 28 12:32:26 relaxed-banach kernel: [6815598.018830] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jul 28 12:32:26 relaxed-banach kernel: [6815598.018947] [3463961]   114 3463961  1201455   173102  4685824   351755             0 mariadbd


Jul 28 12:32:26 relaxed-banach kernel: [6815598.019344] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mariadbd,pid=3463961,uid=114
Jul 28 12:32:26 relaxed-banach kernel: [6815598.019518] Out of memory: Killed process 3463961 (mariadbd) total-vm:4805820kB, anon-rss:692408kB, file-rss:0kB, shmem-rss:0kB, UID:114 pgtables:4576kB oom_score_adj:0
Jul 28 12:32:26 relaxed-banach systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.
Jul 28 12:32:26 relaxed-banach systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL

Jul 25 00:16:25 relaxed-banach mariadbd[3463961]: 2023-07-25  0:16:25 1625222 [Warning] Aborted connection 1625222 to db: 'psa' user: 'admin' host: 'localhost' (Got an error reading communication packets)

 

[Peter Debik]

"Out of memory" means that you need more RAM.