Reputation: 4560
Today I updated openssl (due to the recent heartbleed vulnerability) and all of a sudden mysql is acting strangely. I was recently able to modify tables, but now when I try to add a column I get:
ERROR 1005 (HY000): Can't create table '#sql-34f_872b' (errno: 13)
And trying to do show tables
results in:
ERROR 1018 (HY000): Can't read dir of './<db>/' (errno: 13)
During the openssl upgrade there was a prompt asking about an upgrade to mysql. It was asking if I want to keep my current /etc/my.cnf or if I wanted to replace with the new one - I selected to keep.
Typically this would be a permissions issue, and I've tried and tested the permissions on the mysql datadir
(using this answer from a similar question).
A few other strange things:
I feel like the openssl upgrade is too much of a coincidence to ignore, and I'm not keen on the idea of restarting mysqld without really knowing that the server will definitely come back up (since there's an unknown issue going on here).
Any ideas?
Reply to comments and questions:
Output of ls -ltrFa
:
remy@ip-10-168-9-52:~$ ls -ltrFa /vol/mysql/
total 49367084
-rwxr-xr-x 1 mysql mysql 0 Feb 11 2013 debian-5.5.flag*
drwxr-xr-x 2 mysql mysql 4096 Feb 11 2013 test/
drwxr-xr-x 2 mysql mysql 4096 Feb 11 2013 performance_schema/
drwxr-xr-x 2 mysql mysql 4096 Feb 11 2013 mysql/
-rwxr-xr-x 1 mysql mysql 6 Feb 11 2013 mysql_upgrade_info*
-rwxr-xr-x 1 mysql mysql 25 Feb 12 2013 slave-relay-bin.index*
-rwxr-xr-x 1 mysql mysql 126 Feb 12 2013 slave-relay-bin.000001*
drwxr-xr-x 6 mysql mysql 4096 Oct 14 14:35 ./
drwxr-xr-x 2 mysql mysql 4096 Apr 3 15:50 jsbin/
drwxrwxrwx 5 root root 4096 Apr 9 16:07 ../
-rwxr-xr-x 1 mysql mysql 50417631232 Apr 9 17:24 ibdata1*
-rwxr-xr-x 1 mysql mysql 67108864 Apr 9 17:24 ib_logfile0*
-rwxr-xr-x 1 mysql mysql 67108864 Apr 9 17:25 ib_logfile1*
Output of ps aux | grep mysql
:
mysql 847 1.4 83.9 16342212 14681964 ? Ssl 2013 3646:34 /usr/sbin/mysqld
remy 4038 0.0 0.0 101816 2824 pts/0 S+ 16:58 0:00 mysql -uroot -px xxxxx jsbin
Note that I've also tried running mysql using sudo -u mysql mysql -uroot -pxxx jsbin
and it results in the same issue.
Here is the the log from the apt-get upgrade openssl
which shows mysql being included in the update: https://gist.github.com/remy/10291829
Server is ubuntu-precise-12.04-amd64 (installations are all via apt-get rather than manually compiled).
Versions of mysql:
$ mysqld --version
mysqld Ver 5.5.29-0ubuntu0.12.04.1 for debian-linux-gnu on x86_64 ((Ubuntu))
$ mysql --version
mysql Ver 14.14 Distrib 5.5.35, for debian-linux-gnu (x86_64) using readline 6.2
Upvotes: 10
Views: 13539
Reputation: 107
The following worked for me :
rm /var/lib/mysql/ib_logfile*
/etc/init.d/mysql restart
Upvotes: 5
Reputation: 4560
So it turns out that apt-get upgrade openssl
actually upgrades everything else in it's wake - along with openssl. So somehow MySQL got caught in that update.
By accident (an AWS backup actually) the machine was rebooted, and all problems went away.
I'm pretty sure this is because some of MySQL's libraries had been updated, but the server that was loaded in memory was somehow incompatible.
I wouldn't generally recommend a random restart to fix these issues, but in this particular situation, it did the job.
Upvotes: 2
Reputation: 52
The most-likely reason I can think of for this is due to some upgrade process or script trying to alter table defaults and/or one of the dependent tools adding a table and immediately altering it. This could step on a documented limitation in MySQL that prevents you from adding and dropping foreign keys in the same ALTER statement.: * http://bugs.mysql.com/bug.php?id=68286
Can you check your logs to see what was running when the failure happened and then manually edit the update script or manually make the change so it's skipped?
I think keeping your my.cnf was the right way to go too. But if you can't get enough info from the logs as noted above, you could try disabling FK checks in there, https://dev.mysql.com/doc/refman/5.6/en/create-table-foreign-keys.html
...that does risk some trash data though, so use as a last resort.
As for OpenSSL, this was triggered by the update but not specific to OpenSSL. Any update to 5.5.29 that ran a complex ALTER might have triggered this
Upvotes: 0