Question MySQL wont start after restore

[L. Hagen]

Hi,

two days ago I made a terrible mistake by removing some packages with dependencies to Plesk. So I had to restore from a backup. But unfortunately this fails and MySql wont start. So Plesk isn't staring as well.

My service is hosted by STRATO and the guys told me that when I'm using their recover system that all files (incl. system files) will be restored. So I did (twice) this procedure (with different backups).

When trying to access Plesk I get:

ERROR: Zend_Db_Adapter_Exception: SQLSTATE[HY000] [2002] No such file or directory

Additionally, an exception has occurred while trying to report this error: Zend_Exception
No entry is registered for key 'translate' (Abstract.php:144)

Search for related Knowledge Base articles

I can access the system via SSH and this is running (excepting mysql). When starting the service it sends back a FAIL. The log tells the following:

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x20)[0x7f0523503920]
/usr/sbin/mysqld(handle_fatal_signal+0x3d5)[0x7f05233ecb75]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f0522179330]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f05217cdc37]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f05217d1028]
170607 7:35:03 [Note] Event Scheduler: Loaded 0 events
170607 7:35:03 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.55-0ubuntu0.14.04.1-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
/usr/sbin/mysqld(+0x53cf55)[0x7f052354af55]
/usr/sbin/mysqld(+0x53daa9)[0x7f052354baa9]
/usr/sbin/mysqld(+0x5ff7af)[0x7f052360d7af]
/usr/sbin/mysqld(+0x5f6045)[0x7f0523604045]
/usr/sbin/mysqld(+0x53f915)[0x7f052354d915]
/usr/sbin/mysqld(+0x53151c)[0x7f052353f51c]
/usr/sbin/mysqld(+0x5355e3)[0x7f05235435e3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f0522171184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f0521894bed]
The manual page at MySQL :: MySQL 5.7 Reference Manual :: B.5.3.3 What to Do If MySQL Keeps Crashing contains
information that should help you find out what is causing the crash.
170607 7:35:03 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a futur
170607 7:35:03 [Warning] option 'innodb-buffer-pool-size': signed value 2097152 adjusted to 5242880
170607 7:35:03 [Warning] option 'innodb-additional-mem-pool-size': signed value 512000 adjusted to 524288
170607 7:35:03 [Note] Plugin 'FEDERATED' is disabled.
170607 7:35:03 InnoDB: The InnoDB memory heap is disabled
170607 7:35:03 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170607 7:35:03 InnoDB: Compressed tables use zlib 1.2.8
170607 7:35:03 InnoDB: Using Linux native AIO
170607 7:35:03 InnoDB: Initializing buffer pool, size = 512.0M
170607 7:35:03 InnoDB: Completed initialization of buffer pool
170607 7:35:03 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
170607 7:35:03 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
170607 7:35:03 InnoDB: Waiting for the background threads to start
170607 7:35:04 InnoDB: 5.5.55 started; log sequence number 37278284739
170607 7:35:04 InnoDB: !!! innodb_force_recovery is set to 1 !!!
170607 7:35:04 [Note] Server hostname (bind-address): '::'; port: 3306
170607 7:35:04 [Note] - '::' resolves to '::';
170607 7:35:04 [Note] Server socket created on IP: '::'.
170607 7:35:04 InnoDB: Assertion failure in thread 140310783301376 in file trx0purge.c line 840
InnoDB: Failing assertion: purge_sys->purge_trx_no <= purge_sys->rseg->last_trx_no
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to MySQL Bugs.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: MySQL :: MySQL 5.5 Reference Manual :: 14.23.2 Forcing InnoDB Recovery
InnoDB: about forcing recovery.
05:35:04 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

So I tried a crash recovery without success. I have no idea what else could help. Please does somebody has any help for me?

Thx, Lars

 

[IgorG]

Innodb is corrupted here. Official method for innodb recovery should be used here MySQL :: MySQL 5.7 Reference Manual :: 14.21.2 Forcing InnoDB Recovery
Have you tried this?

 

[L. Hagen]

Hi IgorG,

Yes I did and followed the instructions but without any success :-(

 

[IgorG]

I can only recommend to create a request to support team to do in-depth investigation to find the reason and to fix it. Please create a ticket to support at Help Center

 

[L. Hagen]

Well, I played a bit arround and was able to get plesk running (removing the mysql table and restarting the mysqld). But I had some other problems. So I called the bootstrapper.sh and the plesk repair tool. Both running through without errors. The Plesk system itself is running now without any mistakes. BUT I'm not able to restore my domains. I have four domains on my system. When I restore the entire system after a few minutes the following comes up:

Warning: Restore domain "DOMAINNAME"
Execution of /opt/psa/admin/plib/api-cli/domain.php --update DOMAINNAME -guid a53bb34a-427e-4691-8e04-adcd21378c05 -creation-date 2017-03-06 -description '' -hosting true -hst_type phys -do-not-apply-skeleton -ip IPADDRESS -www-root httpdocs -login USERNAME -passwd '' -passwd_type sym -hard_quota 10737418240B -shell /bin/bash -ignore-nonexistent-options failed with return code 1. Stderr is An error occurred during domain update: An error occurred during changing of hosting settings: The user XXXXX already exists.
Warning: Restore database user "dbuevents"
Failed deployment of database user dbuevents of database dbevents
Warning:
Execution of /opt/psa/admin/plib/api-cli/database.php --update dbevents -add_user dbuevents -server localhost:3306 -type mysql -print-id -default -set-acl 127.0.0.1,IPADDRESS,localhost -set-privileges Select,Insert,Update,Delete,Create,Drop,References,Index,Alter,Create\ Temporary\ Tables,Lock\ Tables,Create\ View,Show\ View,Create\ Routine,Alter\ Routine,Execute,Event,Trigger -passwd '' -passwd_type sym -ignore-nonexistent-options failed with return code 1. Stderr is Unable to create database user: Operation CREATE USER failed for 'XXXXX@'127.0.0.1'

I made this procedure several times and have also checked before that the database was not available anymore and that the username is not in the "psa" and "mysql" databases. A global search returned an ampty result. What I am doing wrong?

Is there a way to get a "clean plesk" installation via SSH and to make the restore on that?

Thanks for your help!!!

 

[Bitpalast]

Looks like the database user that is to be created is missing a password. This does not seem to be a fatal error, but a warning only. Is the database present after the restore attempt? In that case you could simply add a user account to it manually.

 

[L. Hagen]

Looks like the database user that is to be created is missing a password. This does not seem to be a fatal error, but a warning only. Is the database present after the restore attempt? In that case you could simply add a user account to it manually.

Hi Peter,

meanwhile I could got it to work. I could discover a really strange behaviour of Plesk:

1. first of all I cleaned the database completely and removed all user or database entries related to my domains (I also checked the filesystem so that really everything was removed)
2. then I restored a backup with all options
3. this failed with this user exists error
4. then I started again at (1)
5. it failed again but this time with a different user that already existed
6. then I removed the successful restored subscriptions
7. made this several times and always (user already exist, very often a user that was part of before removes subscription - but not always) - even users that I have removed and were no database entry was existing before
8. my last try was to clean up the system again, have restored the system data only and finally restored the subscriptions, only -> this was working

In other words, I guess that this is a bug...