Resolved Can't start mariadb, nor the mysql service.

[bitrogers]

Server operating system version

Ubuntu 18.04 x86_64

Plesk version and microupdate number

18.0.49.2

Hey guys, I hope you're doing well!

root@server:~# journalctl -xe
Aug 06 18:19:12 server grafana-server[1438]: logger=alerting.evalContext t=2023-08-06T18:19:12.85+00
Aug 06 18:19:13 server mysqld[36452]: 2023-08-06 18:19:13 140085128805504 [Note] /usr/sbin/mysqld (m
Aug 06 18:19:13 server systemd[1]: nginx.service: Service hold-off time over, scheduling restart.
Aug 06 18:19:13 server systemd[1]: nginx.service: Scheduled restart job, restart counter is at 474.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: Enterprise open source support | Ubuntu
--
-- Automatic restarting of the unit nginx.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Aug 06 18:19:13 server systemd[1]: Stopped Startup script for nginx service.
-- Subject: Unit nginx.service has finished shutting down
-- Defined-By: systemd
-- Support: Enterprise open source support | Ubuntu
--
-- Unit nginx.service has finished shutting down.
Aug 06 18:19:13 server systemd[1]: Starting Startup script for nginx service...
-- Subject: Unit nginx.service has begun start-up
-- Defined-By: systemd
-- Support: Enterprise open source support | Ubuntu
--
-- Unit nginx.service has begun starting up.
Aug 06 18:19:13 server nginx[36478]: nginx: [emerg] open() "/etc/nginx/plesk.conf.d/vhosts/allhispanos.com.conf" failed (2: No such file or directory) in /etc/nginx/conf.d/zz010_psa_nginx.conf:7
Aug 06 18:19:13 server nginx[36478]: nginx: configuration file /etc/nginx/nginx.conf test failed
Aug 06 18:19:13 server systemd[1]: nginx.service: Control process exited, code=exited status=1
Aug 06 18:19:13 server systemd[1]: nginx.service: Failed with result 'exit-code'.
Aug 06 18:19:13 server systemd[1]: Failed to start Startup script for nginx service.
-- Subject: Unit nginx.service has failed
-- Defined-By: systemd
-- Support: Enterprise open source support | Ubuntu
--
-- Unit nginx.service has failed.
--
-- The result is RESULT.



systemctl status mariadb.service
● mariadb.service - MariaDB 10.1.48 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: activating (start) since Sun 2023-08-06 18:18:10 UTC; 3s ago
Docs: man:mysqld(8)
systemd
Process: 35472 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr
Process: 35470 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exi
Process: 35469 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exit
Main PID: 35564 (mysqld)
Status: "Waiting for page cleaner"
Tasks: 15 (limit: 9484)
CGroup: /system.slice/mariadb.service
└─35564 /usr/sbin/mysqld

Aug 06 18:18:10 server systemd[1]: Starting MariaDB 10.1.48 database server...
Aug 06 18:18:10 server mysqld[35564]: 2023-08-06 18:18:10 140571712470144 [Note] /usr/sbin/mysqld (m
Aug 06 18:18:13 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILU
Aug 06 18:18:13 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 06 18:18:13 server systemd[1]: Failed to start MariaDB 10.1.48 database server.




Mysql log:

2023-08-06 18:12:59 140074868153472 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2023-08-06 18:12:59 140074868153472 [Note] InnoDB: The InnoDB memory heap is disabled
2023-08-06 18:12:59 140074868153472 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-08-06 18:12:59 140074868153472 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2023-08-06 18:12:59 140074868153472 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-08-06 18:12:59 140074868153472 [Note] InnoDB: Using SSE crc32 instructions
2023-08-06 18:12:59 140074868153472 [Note] InnoDB: Initializing buffer pool, size = 5.9G
2023-08-06 18:13:00 140074868153472 [Note] InnoDB: Completed initialization of buffer pool
2023-08-06 18:13:00 140074868153472 [Note] InnoDB: Highest supported file format is Barracuda.
2023-08-06 18:13:00 7f65b8c21c80 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
2023-08-06 18:13:00 140074868153472 [ERROR] InnoDB: Could not find a valid tablespace file for 'admin_all/wp_hostwffilemods'. See MySQL :: MySQL 8.0 Reference Manual :: 15.21.4 Troubleshooting InnoDB Data Dictionary Operations for how to resolve the issue.
2023-08-06 18:13:00 140074868153472 [ERROR] InnoDB: Tablespace open failed for '"admin_all"."wp_hostwffilemods"', ignored.
2023-08-06 18:13:00 7f65b8c21c80 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
2023-08-06 18:13:00 140074868153472 [ERROR] InnoDB: Could not find a valid tablespace file for 'admin_all/wp_hostwfknownfilelist'. See MySQL :: MySQL 8.0 Reference Manual :: 15.21.4 Troubleshooting InnoDB Data Dictionary Operations for how to resolve the issue.
2023-08-06 18:13:00 140074868153472 [ERROR] InnoDB: Tablespace open failed for '"admin_all"."wp_hostwfknownfilelist"', ignored.
2023-08-06 18:13:00 7f65b8c21c80 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
2023-08-06 18:13:00 140074868153472 [ERROR] InnoDB: Could not find a valid tablespace file for 'admin_all/wp_hostwfpendingissues'. See MySQL :: MySQL 8.0 Reference Manual :: 15.21.4 Troubleshooting InnoDB Data Dictionary Operations for how to resolve the issue.
2023-08-06 18:13:00 140074868153472 [ERROR] InnoDB: Tablespace open failed for '"admin_all"."wp_hostwfpendingissues"', ignored.
2023-08-06 18:13:00 140074868153472 [Note] InnoDB: 128 rollback segment(s) are active.
2023-08-06 18:13:00 140074868153472 [Note] InnoDB: Waiting for purge to start
2023-08-06 18:13:00 140074868153472 [Note] InnoDB: Percona XtraDB (Open Source Database Software Support & Services | Percona) 5.6.49-89.0 started; log sequence number 169462526812
2023-08-06 18:13:00 140067532142336 [Note] InnoDB: Dumping buffer pool(s) not yet started
2023-08-06 18:13:00 140074868153472 [Note] Plugin 'FEEDBACK' is disabled.
2023-08-06 18:13:00 140074868153472 [Note] Recovering after a crash using tc.log
2023-08-06 18:13:00 140074868153472 [ERROR] Can't init tc log
2023-08-06 18:13:00 140074868153472 [ERROR] Aborting

 

[AYamshanov]

Just a quick idea to test, do you have a free space on the server? df -h. Because it is quite strange that two independent server have some issues to start (nginx and database).

 

[tkalfaoglu]

hi.. it seems its missing /etc/nginx/plesk.conf.d/vhosts/allhispanos.com.conf file.
so, try

# plesk repair web allhispanos.com

 

[bitrogers]

hi.. it seems its missing /etc/nginx/plesk.conf.d/vhosts/allhispanos.com.conf file.
so, try

# plesk repair web allhispanos.com

root@server:~# plesk repair web allhispanos.com
DB query failed: SQLSTATE[HY000] [2002] No such file or directory

exit status 1
root@server:~#

 

[bitrogers]

Is there a way to delete that project so that everything goes back to normal?

 

[tkalfaoglu]

Can you try
Plesk repair db

If that fails you may have to delete that hosting and recreate it. Make backups...

 

[bitrogers]

Can you try
Plesk repair db

If that fails you may have to delete that hosting and recreate it. Make backups...

How can I export the project database if not I can execute any mysql command?

 

[Peter Debik]

@bitrogers,

You can actually backup your databases if the database server is not running by a full copy of /var/lib/mysql. It is not a database dump in that case, but just in case the path is deleted, it can be restored by moving the copy back into the same path. This requires, though, that the database server version is the same or newer than the one you have been using up to know. This type of saving the data is your last resort. It is not recommended as a backup method. Instead, it will be much better to find the cause why the database server does not start.

- Could you please check whether the path /var/lib/mysql exists on your server and that the mysql directory has permissions 0755?
- Could you please provide the content of your /etc/my.cnf file here?
- Please also provide the mariadb related lines from /var/log/syslog

 

[bitrogers]

Hello Peter, this is the info

 

[Peter Debik]

For the web server, what others said before is good advice, but maybe try again. The cause there is that a symbolic link exists for allhispanos.com.conf (configuration file), but the file itself is missing. This can occur when a domain has been deleted from the server, but the symbolic link was not properly removed for some reason. To fix this:
Make sure that we have a config file set with which Nginx can start for sure.:
# rm /etc/nginx/plesk.conf.d/vhosts/allhispanos.com.conf
Just in case that the domain actually exists and only the file was missing:
# plesk repair web allhispanos.com.conf
Start Nginx:
# systemctl start nginx

For the database issue I meant:
# grep mariadb /var/log/syslog
or
# grep mariadb /var/log/messages
to being able to see the actual messages that MariaDB logs when it tries to start.

 

[bitrogers]

# plesk repair web allhispanos.com.conf

root@server:~# plesk repair web allhispanos.com.conf
DB query failed: SQLSTATE[HY000] [2002] No such file or directory

exit status 1
root@server:~#

root@server:~# grep mariadb /var/log/syslog
Aug 9 14:20:31 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:20:31 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:20:40 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:20:40 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:20:49 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:20:49 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:20:58 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:20:58 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:21:07 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:21:07 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:21:16 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:21:16 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:21:25 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:21:25 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:21:34 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:21:34 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:21:42 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:21:42 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:21:51 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:21:51 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:22:00 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:22:00 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:22:09 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:22:09 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:22:18 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:22:18 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:22:27 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:22:27 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:22:36 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:22:36 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:22:45 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:22:45 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:22:54 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:22:54 server systemd[1]: mariadb.service: Failed with result 'exit-code'.


All content in syslog file -> after run "rm /etc/nginx/plesk.conf.d/vhosts/allhispanos.com.conf"

Aug 9 14:20:27 server systemd[1]: plesk-web-socket.service: Service hold-off time over, scheduling restart.
Aug 9 14:20:27 server systemd[1]: plesk-web-socket.service: Scheduled restart job, restart counter is at 12044.
Aug 9 14:20:27 server systemd[1]: Stopped Plesk Web Socket Service.
Aug 9 14:20:27 server systemd[1]: Starting MariaDB 10.1.48 database server...
Aug 9 14:20:28 server mysqld[82572]: 2023-08-09 14:20:28 139749806075008 [Note] /usr/sbin/mysqld (mysqld 10.1.48-MariaDB-0ubuntu0.18.04.1) starting as process 82572 ...
Aug 9 14:20:31 server grafana-server[1477]: logger=alerting.evalContext t=2023-08-09T14:20:31.05+0000 lvl=eror msg="Alert Rule Result Error" ruleId=1 name="Apache & PHP-FPM memory usage" error="request handler response error {invalid status code. status: 500 Internal Server Error A <nil> [] [] 0xc000610000}" changingstateto=keep_state
Aug 9 14:20:31 server systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:20:31 server systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 9 14:20:31 server systemd[1]: Failed to start MariaDB 10.1.48 database server.
Aug 9 14:20:31 server systemd[1]: Started Plesk Web Socket Service.
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: [2023-08-09 14:20:31.413] 82597:64d3a0af64c32 ERR [panel] DB query failed: SQLSTATE[HY000] [2002] No such file or directory:
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: 0: /opt/psa/admin/plib/Db/Adapter/Pdo/Mysql.php:79
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: #011Db_Adapter_Pdo_Mysql->query(string 'SET sql_mode = ''')
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: 1: /opt/psa/admin/plib/CommonPanel/Application/Abstract.php:103
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: #011CommonPanel_Application_Abstract::initDbAdapter()
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: 2: /opt/psa/admin/plib/Session/Helper.php:176
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: #011Plesk\Session\Helper::initStorage()
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: 3: /opt/psa/admin/plib/CommonPanel/Application/Abstract.php:52
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: #011CommonPanel_Application_Abstract->run()
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: 4: /opt/psa/admin/plib/CommonPanel/Application/Abstract.php:34
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: #011CommonPanel_Application_Abstract::init()
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: 5: /opt/psa/admin/plib/pm/Bootstrap.php:16
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: #011pm_Bootstrap::init()
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: 6: /opt/psa/admin/plib/sdk.php:11
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: #011require_once(string '/opt/psa/admin/plib/sdk.php')
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: 7: /opt/psa/admin/plib/WebSocket/bin/ws-server.php:3
Aug 9 14:20:31 server sw-engine-pleskrun[82597]: ERROR: Plesk\Exception\Database: DB query failed: SQLSTATE[HY000] [2002] No such file or directory (Mysql.php:79)
Aug 9 14:20:31 server systemd[1]: plesk-web-socket.service: Main process exited, code=exited, status=1/FAILURE
Aug 9 14:20:31 server systemd[1]: plesk-web-socket.service: Failed with result 'exit-code'.
Aug 9 14:20:32 server grafana-server[1477]: logger=alerting.evalContext t=2023-08-09T14:20:32.05+0000 lvl=eror msg="Alert Rule Result Error" ruleId=6 name="Partition \"/\" utilization" error="request handler response error {invalid status code. status: 500 Internal Server Error A <nil> [] [] 0xc000c74540}" changingstateto=keep_state
Aug 9 14:20:33 server grafana-server[1477]: logger=alerting.evalContext t=2023-08-09T14:20:33.05+0000 lvl=eror msg="Alert Rule Result Error" ruleId=4 name="MySQL memory usage" error="request handler response error {invalid status code. status: 500 Internal Server Error A <nil> [] [] 0xc0011c8a80}" changingstateto=keep_state
Aug 9 14:20:33 server grafana-server[1477]: logger=alerting.evalContext t=2023-08-09T14:20:33.06+0000 lvl=eror msg="Alert Rule Result Error" ruleId=2 name="nginx memory usage" error="request handler response error {invalid status code. status: 500 Internal Server Error A <nil> [] [] 0xc0011c9bf0}" changingstateto=keep_state
Aug 9 14:20:36 server grafana-server[1477]: logger=alerting.evalContext t=2023-08-09T14:20:36.05+0000 lvl=eror msg="Alert Rule Result Error" ruleId=9 name="Swap usage" error="request handler response error {invalid status code. status: 500 Internal Server Error A <nil> [] [] 0xc0006c6360}" changingstateto=keep_state
Aug 9 14:20:36 server grafana-server[1477]: logger=alerting.evalContext t=2023-08-09T14:20:36.06+0000 lvl=eror msg="Alert Rule Result Error" ruleId=3 name="Mail server memory usage" error="request handler response error {invalid status code. status: 500 Internal Server Error A <nil> [] [] 0xc001ab4ea0}" changingstateto=keep_state
Aug 9 14:20:36 server grafana-server[1477]: logger=alerting.evalContext t=2023-08-09T14:20:36.06+0000 lvl=eror msg="Alert Rule Result Error" ruleId=5 name="Plesk memory usage" error="request handler response error {invalid status code. status: 500 Internal Server Error A <nil> [] [] 0xc0006c78f0}" changingstateto=keep_state
Aug 9 14:20:36 server systemd[1]: plesk-web-socket.service: Service hold-off time over, scheduling restart.
Aug 9 14:20:36 server systemd[1]: plesk-web-socket.service: Scheduled restart job, restart counter is at 12045.
Aug 9 14:20:36 server systemd[1]: Stopped Plesk Web Socket Service.
Aug 9 14:20:36 server systemd[1]: Starting MariaDB 10.1.48 database server...
Aug 9 14:20:37 server mysqld[82693]: 2023-08-09 14:20:37 140116205644928 [Note] /usr/sbin/mysqld (mysqld 10.1.48-MariaDB-0ubuntu0.18.04.1) starting as process 82693 ...

 

[Peter Debik]

Unfortunately I've never before seen a MariaDB server that crashes without giving any hint why. You did check that you have enough disk space, didn't you?

 

[bitrogers]

Hello I found a solution fo this problem, My server It's running now, Thank you @Peter Debik @tkalfaoglu @AYamshanov for help me.

The solution was:

Go to: /var/lib/mysql/
Remove this file: tc.log