docker mariadb server won't start after server downtime

Question

my server was down for a few hours and when the datacenter was back online i couldnt start my database sever, i do not know if i can do anything to recovery any of my databases, and any help would be appreciated. This is the error log i am receiving using docker logs, i tried everything i can think of, please help.

       InnoDB: about forcing recovery.
180503 15:09:51 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.3.4-MariaDB-10.3.4+maria~jessie
key_buffer_size=134217728
read_buffer_size=2097152
max_used_connections=0
max_threads=102
thread_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759982 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f052c0009a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f053bffed78 thread_stack 0x49000
mysqld(my_print_stacktrace+0x2e)[0x55794acdcafe]
mysqld(handle_fatal_signal+0x347)[0x55794a755a27]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f0593cfd890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f0592088067]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f0592089448]
mysqld(+0x4af752)[0x55794a4a3752]
mysqld(+0xa1bb35)[0x55794aa0fb35]
mysqld(+0xa1cd42)[0x55794aa10d42]
mysqld(+0xa1d8ce)[0x55794aa118ce]
mysqld(+0xa014a0)[0x55794a9f54a0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7f0593cf6064]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f059213b62d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0):
Connection ID (thread ID): 2
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Fatal signal 11 while backtracing
2018-05-03 15:10:40 0 [Note] mysqld (mysqld 10.3.4-MariaDB-10.3.4+maria~jessie) starting as process 1 ...
2018-05-03 15:10:40 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-05-03 15:10:40 0 [Note] InnoDB: Uses event mutexes
2018-05-03 15:10:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-05-03 15:10:40 0 [Note] InnoDB: Using Linux native AIO
2018-05-03 15:10:40 0 [Note] InnoDB: Number of pools: 1
2018-05-03 15:10:40 0 [Note] InnoDB: Using SSE2 crc32 instructions
2018-05-03 15:10:40 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2018-05-03 15:10:40 0 [Note] InnoDB: Completed initialization of buffer pool
2018-05-03 15:10:40 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2018-05-03 15:10:40 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=60464127549
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=823] log sequence number 60464206926 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=384] log sequence number 60464207004 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=385] log sequence number 60464207757 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=388] log sequence number 60464208763 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=389] log sequence number 60464210321 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=410] log sequence number 60464214175 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=192] log sequence number 60464214258 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=193] log sequence number 60464214302 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=194] log sequence number 60464214346 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=195] log sequence number 60464214390 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=196] log sequence number 60464214434 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=197] log sequence number 60464214478 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=198] log sequence number 60464215658 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=199] log sequence number 60464215702 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=200] log sequence number 60464215746 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=401] log sequence number 60464215746 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=201] log sequence number 60464215790 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=202] log sequence number 60464216128 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=203] log sequence number 60464216397 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=204] log sequence number 60464216659 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=205] log sequence number 60464217025 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=407] log sequence number 60464201860 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=391] log sequence number 60464201904 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=586] log sequence number 60464197776 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=392] log sequence number 60464202212 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=0, page number=821] log sequence number 60464207440 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [Note] InnoDB: Starting final batch to recover 70 pages from redo log.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=607, page number=28013] log sequence number 60464199864 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=607, page number=3] log sequence number 60464199832 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Page [page id: space=607, page number=27055] log sequence number 60464199929 is in the future! Current system log sequence number 60464197030.
2018-05-03 15:10:40 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/ for information about forcing recovery.
2018-05-03 15:10:41 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2018-05-03 15:10:41 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2018-05-03 15:10:41 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2018-05-03 15:10:41 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2018-05-03 15:10:41 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2018-05-03 15:10:41 0 [Note] InnoDB: Waiting for purge to start
2018-05-03 15:10:41 0x7f57917fa700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.3.4/storage/innobase/include/fut0lst.ic line 85
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/
InnoDB: about forcing recovery.
180503 15:10:41 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.3.4-MariaDB-10.3.4+maria~jessie
key_buffer_size=134217728
read_buffer_size=2097152
max_used_connections=0
max_threads=102
thread_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759982 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f57800009a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f57917f9d78 thread_stack 0x49000
mysqld(my_print_stacktrace+0x2e)[0x558c4f0cbafe]
mysqld(handle_fatal_signal+0x347)[0x558c4eb44a27]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f57e1033890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f57df3be067]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f57df3bf448]
mysqld(+0x4af752)[0x558c4e892752]
mysqld(+0xa1bb35)[0x558c4edfeb35]
mysqld(+0xa1cd42)[0x558c4edffd42]
mysqld(+0xa1d8ce)[0x558c4ee008ce]
mysqld(+0xa014a0)[0x558c4ede44a0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7f57e102c064]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f57df47162d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0):
Connection ID (thread ID): 2
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Fatal signal 11 while backtracing

Answer 1

i tried everything but the only solution that worked for me in case someone else needs help is using MySQL Utilities. first i backed up the var/lib/mysql into /backup/db folder then i run this: mysqlfrm --server=root:ROOTPASS@host DATABASE_NAME:TABLENAME.frm --port=3307 --user=root , the command will return the CREATE TABLE statement . then in a newly installed database i run the create table. After that i run ALTER TABLE mydb.mytable DISCARD TABLESPACE; copy mytable.ibd using cp -ap /home/backup/DATABASE_NAME/ mytable.ibd /var/lib/mysql/DATABASE_NAME then i run ALTER TABLE mydb.mytable IMPORT TABLESPACE; you have to repeate the process for each table and it finally worked. I hope this is helpful

Answer 2

You should use Innodb force recovery mode to start db, backup your data. After backup, you should clean your database file switch db to normal mode (remove innodb_force_recovery) and then restore database

For innodb recovery mode refer https://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html .Remember, in recovery mode you can read data only and cannot update, insert or delete so you should backup your database only

Detail step: find your my.ini (on linux my.cnf) add this line to your [mysql]

innodb_force_recovery = 1

Try to increase it from 1 -> 6 until you can start your database server