MariaDB 10 CentOS 7 moving datadir woes

Question

Brand new "minimal" install of CentOS 7 along with MariaDB 10. I have an additional mounted mirrored volume that I want to use for the datadir. Startup sequence is fine and completes normally when my.cnf [mysqld] is commented out. I've copied the data..

sudo cp -R -p /var/lib/mysql/* /mnt/mysql/

The permissions are identical to those of the original. The volume is in /etc/fstab and mounts fine /dev/sdb1 /mnt/mysql xfs defaults 0 0

[root@femur mysql]# ls -la
total 110632
drwxr-xr-x. 5 mysql mysql     4096 Oct 20 15:27 .
drwxr-xr-x. 3 root  root        18 Oct 16 16:46 ..
-rw-rw----. 1 mysql mysql    16384 Oct 20 15:27 aria_log.00000001
-rw-rw----. 1 mysql mysql       52 Oct 20 15:27 aria_log_control
-rw-r-----. 1 mysql root      7005 Oct 20 13:49 femur.err
-rw-rw----. 1 mysql mysql 12582912 Oct 20 15:27 ibdata1
-rw-rw----. 1 mysql mysql 50331648 Oct 20 15:27 ib_logfile0
-rw-rw----. 1 mysql mysql 50331648 Oct 20 12:21 ib_logfile1
-rw-rw----. 1 mysql mysql        0 Oct 20 12:22 multi-master.info
drwx--x--x. 2 mysql mysql     4096 Oct 20 12:21 mysql
drwx------. 2 mysql mysql     4096 Oct 20 13:37 performance_schema
drwxr-xr-x. 2 mysql mysql        6 Oct 20 12:21 test

this is in my.cnf

!includedir /etc/my.cnf.d
[mysqld]
log_error = /var/log/mysql-error.log
user = mysql
datadir = /mnt/mysql
socket = /mnt/mysql/mysql.sock

This is what I get when I try to start it...

'[root@femur mysql]# sudo systemctl start mysql.service
Job for mysql.service failed. See 'systemctl status mysql.service' and 'journalctl -xn' for details.'

Neither of those two files says much, but this is in /var/log/mysql-error.log

141020 16:07:09 mysqld_safe Starting mysqld daemon with databases from /mnt/mysql
141020 16:07:09 [Warning] Can't create test file /mnt/mysql/femur.lower-test
141020 16:07:09 [Note] InnoDB: Using mutexes to ref count buffer pool pages
141020 16:07:09 [Note] InnoDB: The InnoDB memory heap is disabled
141020 16:07:09 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
141020 16:07:09 [Note] InnoDB: Memory barrier is not used
141020 16:07:09 [Note] InnoDB: Compressed tables use zlib 1.2.7
141020 16:07:09 [Note] InnoDB: Using Linux native AIO
141020 16:07:09 [Note] InnoDB: Using CPU crc32 instructions
141020 16:07:09 [Note] InnoDB: Initializing buffer pool, size = 128.0M
141020 16:07:09 [Note] InnoDB: Completed initialization of buffer pool
2014-10-20 16:07:09 7f6cb59c9880  InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
141020 16:07:09 [ERROR] InnoDB: os_file_get_status() failed on './ibdata1'. Can't determine file permissions
141020 16:07:09 [ERROR] InnoDB: The system tablespace must be writable!
141020 16:07:09 [ERROR] Plugin 'InnoDB' init function returned error.
141020 16:07:09 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
141020 16:07:09 [ERROR] mysqld: File '/mnt/mysql/aria_log_control' not found (Errcode: 13 "Permission denied")
141020 16:07:09 [ERROR] mysqld: Got error 'Can't open file' when trying to use aria control file '/mnt/mysql/aria_log_control'
141020 16:07:09 [ERROR] Plugin 'Aria' init function returned error.
141020 16:07:09 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
141020 16:07:09 [Note] Plugin 'FEEDBACK' is disabled.
141020 16:07:09 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
141020 16:07:09 [ERROR] Unknown/unsupported storage engine: InnoDB
141020 16:07:09 [ERROR] Aborting

141020 16:07:09 [Note] /usr/sbin/mysqld: Shutdown complete

141020 16:07:09 mysqld_safe mysqld from pid file /mnt/mysql/femur.pid ended

http://www.reddit.com/r/linuxadmin/comments/2ebhpf/adventures_in_moving_mariadb_data_folder/ helped a bit, but I wasn't able to get it to work.

Any help would be greatly appreciated.

Answer 1

I found this step by step guide working for me: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/SELinux_Users_and_Administrators_Guide/sect-Managing_Confined_Services-MariaDB-Configuration_Examples.html

You must install: yum install policycoreutils-python

Guide:

View the SELinux context of the default database location for mysql:

~]# ls -lZ /var/lib/mysql
drwx------. mysql mysql system_u:object_r:mysqld_db_t:s0 mysql

This shows mysqld_db_t which is the default context element for the location of database files. This context will have to be manually applied to the new database location that will be used in this example in order for it to function properly.

Stop the mysqld daemon:

~]# systemctl stop mariadb.service

Create a new directory for the new location of the database(s). In this example, /mysql/ is used:

~]# mkdir -p /mysql

Copy the database files from the old location to the new location:

~]# cp -R /var/lib/mysql/* /mysql/

Change the ownership of this location to allow access by the mysql user and group. This sets the traditional Unix permissions which SELinux will still observe:

~]# chown -R mysql:mysql /mysql

Run the following command to see the initial context of the new directory:

~]# ls -lZ /mysql
drwxr-xr-x. mysql mysql unconfined_u:object_r:usr_t:s0   mysql

The context usr_t of this newly created directory is not currently suitable to SELinux as a location for MariaDB database files. Once the context has been changed, MariaDB will be able to function properly in this area.

Open the main MariaDB configuration file /etc/my.cnf with a text editor and modify the datadir option so that it refers to the new location. In this example the value that should be entered is /mysql:

[mysqld]
datadir=/mysql

Save this file and exit.

Start mysqld. The service should fail to start, and a denial message will be logged to the /var/log/messages file:

~]# systemctl start mariadb.service

Job for mariadb.service failed. See 'systemctl status postgresql.service' and 'journalctl -xn' for details.

However, if the audit daemon is running and with him the setroubleshoot service, the denial will be logged to the /var/log/audit/audit.log file instead: SELinux is preventing /usr/libexec/mysqld "write" access on /mysql. For complete SELinux messages. run sealert -l b3f01aff-7fa6-4ebe-ad46-abaef6f8ad71

The reason for this denial is that /mysql/ is not labeled correctly for MariaDB data files. SELinux is stopping MariaDB from having access to the content labeled as usr_t. Perform the following steps to resolve this problem:

Run the following command to add a context mapping for /mysql/. Note that the semanageutility is not installed by default. If it missing on your system, install the policycoreutils-pythonpackage.

**~]# semanage fcontext -a -t mysqld_db_t "/mysql(/.*)?"**

This mapping is written to the /etc/selinux/targeted/contexts/files/file_contexts.local file:

~]# grep -i mysql /etc/selinux/targeted/contexts/files/file_contexts.local

/mysql(/.*)? system_u:object_r:mysqld_db_t:s0

Now use the restorecon utility to apply this context mapping to the running system:

**~]# restorecon -R -v /mysql**

Now that the /mysql/ location has been labeled with the correct context for MariaDB, mysqldstarts:

~]# systemctl start mariadb.service

Confirm the context has changed for /mysql/:

~]$ ls -lZ /mysql
drwxr-xr-x. mysql mysql system_u:object_r:mysqld_db_t:s0 mysql

The location has been changed and labeled, and mysqld has started successfully. At this point all running services should be tested to confirm normal operation.

Answer 2

The issue is indeed SELinux; you need to do three things before MariaDB / MySQL will start on CentOS 7:

1. Ensure the user:group is mysql:mysql
2. Set the SELinux tag to mysqld_db_t
3. Set the SELinux user to system_u

This is as simple as:

chcon -Rt mysqld_db_t /database/db
chcon -Ru system_u /database/db
chown -R mysql:mysql /database/db

The whole thing I needed to do after plugging in a disk is below:

cfdisk /dev/sdb
pvcreate /dev/sdb1
vgcreate database /dev/sdb1
lvcreate -l 100%FREE -n db database
mkfs.ext4 /dev/database/db
mkdir /database
mount /database
mkdir /database/db
chcon -Rt mysqld_db_t /database/db
chcon -Ru system_u /database/db
chown -R mysql:mysql /database/db
systemctl start mariadb

Answer 3

Well that was interesting...

It turns out, that CentOS 7 "minimal" installs SELinux, which apparently was preventing mysql from writing to the mounted mirrored set. I was looking for security items that I might not have thought about and found it right there in the docs. I had previously thought (obviously erroneously) that SELinux was a distribution, not a module. Once I ran the test to see if it was there....

getenforce

I temporarily disabled it to test.

setenforce 0

Finally, I was able to start MariaDB with the directory in the mirrored set as the datadir and no errors. To make this change permanent (because this server is behind a firewall), in /etc/selinux/config, I made

- SELINUX=enforcing
+ SELINUX=disabled

I hope this helps someone else. Have a great day!