CODEWITHSUNDEEP
mysqlamazon-web-servicesamazon-rds
Sajeeva Lakmal
Sajeeva Lakmal

Reputation: 197

AWS MySQL RDS instance becomes unresponsive and getting restarted automatically

We have a AWS MySQL RDS instance which is about 1.7T in size. Sometimes it becomes unresponsive and no operations can be performed.

  1. CPU utilization, Write IOPS, read IOPS, queue depth, write throughput, write latency and read latency drops to zero.
  2. Number of connections get piled up.
  3. "Show engine innodb status" hangs

  4. Lot of queries (around 25 for each) by rdsadmin which are in hang state.

    SELECT count(*) from mysql.rds_replication_status WHERE action = 'reset slave' and master_host is NULL and master_port is NULL GROUP BY action_timestamp,called_by_user,action,mysql_version,master_host,master_port ORDER BY action_timestamp LIMIT 1;

SELECT NAME, VALUE FROM mysql.rds_configuration;

  5. After sometime, instance gets rebooted automatically with following error.

    MySQL restart initiated to address MySQL induced log backup issues. Note that as part of this resulution, a DB Snapshot will be performed after MySQL completes restarting.

enter image description here

What can be the issue? This happens quite often. Sometimes, for our surprise, this happens in off-peak times too.

Upvotes: 3

Views: 3564

Answers (3)

Sajeeva Lakmal
Sajeeva Lakmal

Reputation: 197

We were able to resolve this issue by upgrading the instances to 5.6.34.

Upvotes: 0

Anshul
Anshul

Reputation: 806

I faced the same issue and raised an issue with AWS Support. Got the following explanation:

The RDS Monitoring service discovered issue regarding backing up Binary Logs of your databases which is critical for Point in Time Restore (PITR) feature. To mitigate this issue and in order to avoid data corruption, RDS monitoring restarted the RDS instance and hence a restart was automatically triggered. In order to make sure that there is no data loss it took a snapshot of DB instance.

Although the RDS instance was multi-AZ it didn't fail over because of following reason:

Multi AZ has 2 criteria: 1- Single Box Experience, which means that Customer always finds his data even after failover. 2- Higher Availability than Single AZ.

So both criteria have to be present when AWS monitoring service takes the Decision to failover to the standby instance, but in your case AWS monitoring service noticed some risk that can cause data loss after the failover and that is why it took the decision to reboot instead of failing over.

Hope this helps. This has happened to me 3 times in last one week though.

Upvotes: 1

Biswajit Mohanty
Biswajit Mohanty

Reputation: 413

check your db maintenance window timing i mean when your schedule maintenance is happening , and note at what time this issue happening is it happening in regular interval or randomly .

check both mysql error logs and slow query logs.

if possible paste the suspected issue here

Upvotes: 0

Related Questions