MySQL eating 80-100% CPU starting July 1 Midnight, a day or so following a EBS connectivity issue

Question

I am extremely curious about a recent issue that I experienced with all of my EC2 instances running MySQL. On July 1st at 12:00 AM UTC, both of my Amazon EC2 instances (running in US-EAST region) fired an alarm indicating high CPU usage. I investigated to find that it was MySQL that was eating all of the CPU. I logged in and ran SHOW PROCESSLIST to find that no queries were running (these servers don't get hit much after business hours). I stopped MySQL, CPU utilization dropped back down to 1-3% (as normal). I restarted MySQL, and it started eating a lot of CPU again. Then, I restarted the server shutdown -r now, and the problem went away. Both servers had the exact same problem (running Ubuntu 12.04 LTS).

The only relevant item I see in the syslog is:

Jun 30 23:59:59 hostname kernel: [14152976.187987] Clock: inserting leap second 23:59:60 UTC

I don't think that I have logging enabled for MySQL; whatever the default setting is for Ubuntu 12.04 is what I'm using. If it makes any difference, most of the databases are InnoDB.

A day prior, the one instance was affected by a EBS connectivity issue, which caused the server to respond extremely slowly when accessing the disk. I don't believe the other instance was effected (it's in another AWS account and responded okay during the "outage"), but I can't be 100% sure. I wonder if this might have caused MySQL to go crazy? But why wait until midnight? (Watch out! This might be the "red herring" in this mystery)

Any thoughts on what might have caused this issue?

Answer 1

Check out this. I think that's the problem you are experiencing.

Answer 2

I believe the cause is the leap second bug. You can find posts about that everywhere :)

Read more here or on serverfault

A quick solutions is to stop the ntpd daemon, synchronize the clock and restart the daemon. Of course some system update won't hurt :)