Duration of starting Mysqld is too long on Lustre file system (InnoDB: Unable to lock ./ibdata1, error: 38)

Question

I can start mysqld and use it normally. But the duration of start is very long (over 3 minuts). When I check log file (/var/log/mysqld.log), I found InnoDB: Unable to lock ./ibdata1, error: 38.

Recently, I moved my mysql data from /var/lib/mysql to /home/user/mysql because files are too large. Then I changed datadir in /etc/my.cnf and /etc/init.d/mysqld respectively and changed datasock in etc/my.cnf. Owner and mod of /home/user/mysql and files in it is correctly set as well.

The /home volume in this installation is on the Lustre file system.

When I found there are 3 innodb engine table in my database, I droped them. But the problem is still here.

Here is the log when starting mysqld (/var/log/mysqld.log).

141027 19:40:03 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
141027 19:40:04 mysqld_safe Starting mysqld daemon with databases from /home/user/mysql
InnoDB: Unable to lock ./ibdata1, error: 38
141027 19:40:04  InnoDB: Retrying to lock the first data file
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
141027 19:41:45  InnoDB: Unable to open the first data file
InnoDB: Error in opening ./ibdata1
141027 19:41:45  InnoDB: Operating system error number 38 in a file operation.
InnoDB: Error number 38 means 'Function not implemented'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not InnoDB: remove old data files which contain your precious data!
141027 19:41:45 [ERROR] Plugin 'InnoDB' init function returned error.
141027 19:41:45 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
141027 19:41:45 [Note] Event Scheduler: Loaded 0 events
141027 19:41:45 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.52'  socket: '/home/user/mysql/mysql.sock'  port: 3306  Source distribution

Answer 1

You're using a Lustre file system for your /home volume. Unless you enable it specifically on a Lustre volume, that file system doesn't support file locking. But InnoDB needs to lock its files to function correctly. So when InnoDB issues the ioctl(2) calls to lock the file, the file system kicks back "Function not implemented." That's what your error log says.

InnoDB: Error in opening ./ibdata1
141027 19:41:45  InnoDB: Operating system error number 38 in a file operation.
InnoDB: Error number 38 means 'Function not implemented'.

Here's a link to an old listserv on this point. http://lists.lustre.org/pipermail/lustre-discuss/2007-August/003768.html Somewhere in that thread somebody mentions how to enable file locking.

You may want to work with the people administering your file systems to create a special-purpose Lustre volume for your MySQL database, a volume that has locking enabled.

Is this a mission-critical MySQL server? If so, please do a lot of due-diligence before deploying it on this file system.