Reputation: 43
After a power failure on my hypervisor, starting up my Zabbix VM appliance, i am unable to get MySQL 8.0 to start.
When I check the status of mysqld I get:
[root@appliance my.cnf.d]# systemctl status mysqld.service
● mysqld.service - MySQL 8.0 database server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/mysqld.service.d
└─override.conf
Active: activating (start) since Thu 2023-05-11 02:02:03 UTC; 4min 0s ago
Process: 4089 ExecStopPost=/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS)
Process: 4144 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mysqld.service (code=exited, status=0/SUCCESS)
Process: 4118 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Main PID: 4182 (mysqld)
Status: "Server startup in progress"
Tasks: 20 (limit: 24872)
Memory: 456.0M
CGroup: /system.slice/mysqld.service
└─4182 /usr/libexec/mysqld --basedir=/usr
May 11 02:02:03 appliance systemd[1]: Starting MySQL 8.0 database server...
May 11 02:02:04 appliance mysql-check-socket[4118]: Socket file /var/lib/mysql/mysql.sock exists.
May 11 02:02:04 appliance mysql-check-socket[4118]: No process is using /var/lib/mysql/mysql.sock, which means it is a garbage, so it will be rem>
...skipping...
PLenty of space:
[root@appliance mysql]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 2.0G 17M 1.9G 1% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/sda2 4.0G 2.3G 1.8G 56% /
/dev/sda3 1014M 40M 975M 4% /tmp
/dev/sda1 488M 136M 316M 31% /boot
/dev/sda5 15G 4.1G 11G 29% /var/lib/mysql
tmpfs 393M 0 393M 0% /run/user/0
When trying to run mysql from cli:
[root@appliance mysql]# mysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
I have checked the mysqld.log:
2023-05-10T09:10:05.988685Z 0 [Warning] [MY-010091] [Server] Can't create test file /var/lib/mysql/mysqld_tmp_file_case_insensitive_test.lower-test
2023-05-10T09:10:06.023830Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2023-05-10T09:10:06.025038Z 0 [System] [MY-010116] [Server] /usr/libexec/mysqld (mysqld 8.0.30) starting as process 1178
2023-05-10T09:10:06.209581Z 0 [Warning] [MY-010091] [Server] Can't create test file /var/lib/mysql/mysqld_tmp_file_case_insensitive_test.lower-test
2023-05-10T09:10:06.209629Z 0 [Warning] [MY-010159] [Server] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
mysqld: File './1.~rec~' not found (OS errno 28 - No space left on device)
2023-05-10T09:10:06.324694Z 0 [ERROR] [MY-010846] [Server] MYSQL_BIN_LOG::open_purge_index_file failed to open register file.
2023-05-10T09:10:06.324746Z 0 [ERROR] [MY-010817] [Server] MYSQL_BIN_LOG::open_index_file failed to sync the index file.
2023-05-10T09:10:06.325034Z 0 [ERROR] [MY-010119] [Server] Aborting
2023-05-10T09:10:06.351653Z 0 [System] [MY-010910] [Server] /usr/libexec/mysqld: Shutdown complete (mysqld 8.0.30) Source distribution.
2023-05-10T09:26:27.393946Z 0 [Warning] [MY-010091] [Server] Can't create test file /var/lib/mysql/mysqld_tmp_file_case_insensitive_test.lower-test
2023-05-10T09:26:27.419508Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2023-05-10T09:26:27.421077Z 0 [System] [MY-010116] [Server] /usr/libexec/mysqld (mysqld 8.0.30) starting as process 1244
2023-05-10T09:26:27.514061Z 0 [Warning] [MY-010091] [Server] Can't create test file /var/lib/mysql/mysqld_tmp_file_case_insensitive_test.lower-test
2023-05-10T09:26:27.514077Z 0 [Warning] [MY-010159] [Server] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
mysqld: File './1.~rec~' not found (OS errno 28 - No space left on device)
2023-05-10T09:26:27.607875Z 0 [ERROR] [MY-010846] [Server] MYSQL_BIN_LOG::open_purge_index_file failed to open register file.
2023-05-10T09:26:27.607905Z 0 [ERROR] [MY-010817] [Server] MYSQL_BIN_LOG::open_index_file failed to sync the index file.
2023-05-10T09:26:27.608322Z 0 [ERROR] [MY-010119] [Server] Aborting
2023-05-10T09:26:27.623067Z 0 [System] [MY-010910] [Server] /usr/libexec/mysqld: Shutdown complete (mysqld 8.0.30) Source distribution.
2023-05-10T09:33:44.681521Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2023-05-10T09:33:44.682379Z 0 [System] [MY-010116] [Server] /usr/libexec/mysqld (mysqld 8.0.30) starting as process 1227
2023-05-10T09:33:44.892304Z 0 [Warning] [MY-013907] [InnoDB] Deprecated configuration parameters innodb_log_file_size and/or innodb_log_files_in_group have been used to compute innodb_redo_log_capacity=536870912. Please use innodb_redo_log_capacity instead.
2023-05-10T09:33:44.995257Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-05-10T09:33:47.626844Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-05-10T09:33:52.151479Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
2023-05-10T09:33:54.171093Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
2023-05-10T09:33:55.430957Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2023-05-10T09:33:55.431023Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2023-05-10T09:33:55.624686Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/lib/mysql/mysqlx.sock
2023-05-10T09:33:55.624913Z 0 [System] [MY-010931] [Server] /usr/libexec/mysqld: ready for connections. Version: '8.0.30' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution.
2023-05-10T14:51:56.999606Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2023-05-10T14:51:57.003019Z 0 [System] [MY-010116] [Server] /usr/libexec/mysqld (mysqld 8.0.30) starting as process 1245
Edit new mysqld.log logs:
2023-05-11T03:32:10.936052Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2023-05-11T03:32:10.936081Z 0 [System] [MY-010116] [Server] /usr/libexec/mysqld (mysqld 8.0.30) starting as process 5316
2023-05-11T03:32:10.944687Z 0 [Warning] [MY-013907] [InnoDB] Deprecated configuration parameters innodb_log_file_size and/or innodb_log_files_in_group have been used to compute innodb_redo_log_capacity=536870912. Please use innodb_redo_log_capacity instead.
2023-05-11T03:32:10.946878Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-05-11T03:32:11.449661Z 0 [ERROR] [MY-011906] [InnoDB] Database page corruption on disk or a failed file read of page [page id: space=71, page number=10344]. You may have to recover from a backup.
Permissions on the socket:
[root@appliance my.cnf.d]# ls -l /var/run/mysqld/mysqld.sock
-rw-r--r-- 1 mysql mysql 0 May 10 22:58 /var/run/mysqld/mysqld.sock
I have tried: mysqld --tc-heuristic-recover=ROLLBACK
Rebooting the server. Tied resetting permissions on mysql instance:
sudo chown mysql:mysql /var/run/mysqld/mysqld.sock
No such luck, is there anything I am overlooking? Thanks in advance
Upvotes: 0
Views: 1039