CODEWITHSUNDEEP
amazon-web-servicesamazon-rdsamazon-aurora
ErnieAndBert
ErnieAndBert

Reputation: 1702

Restore Aurora DB cluster from S3 not working

I am attempting to convert an RDS snapshot to S3 then restore Aurora DB cluster from S3 . Before I commit to the process and convert some of our production snapshots I am testing converting and restoring them back to a DB and having issues. Here is where I am at and the steps taken :

1 - I took a snapshot of a current running Aurora DB - Engine version 5.7.mysql_aurora.2.09.1

2 - After the snapshot was created I selected it and chose ""Export to Amazon S3". I selected to export all of the data.

3 - After about a day it completed the export to S3. It is a 4TB snapshot so it took a while. Even after it was complete I waited two days before I performed the next steps - just to make sure all was complete.

4 - In the RDS dashboard I clicked on "Restore Aurora DB cluster from S3". a - I pointed it to the S3 bucket that the snapshot exported to in step 2. b - I chose to have it create an IAM role that allows it access to this S3 location. c - I chose Aurora as the type and selected the same engine that it used when the DB was originally running and the same VPC and other parameters the original was using.

5 - I clicked on create and after a few moments I get the error : We're sorry, your request to create DB instance database-1 has failed. Files from the specified Amazon S3 bucket cannot be downloaded. Make sure that you have created an AWS Identity and Access Management (IAM) role that lets Amazon RDS access Amazon S3 for you.

6 - I modified the IAM role that was created to to give it all pretty mush admin S3 permissions, reran the "Restore Aurora DB cluster from S3" and still get the same error.

The process seems pretty simple - what could I possibly be doing wrong?

Here is the json of the IAM role after I modified it to allow it access to all S3 -

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Upvotes: 0

Views: 3166

Answers (2)

Larry
Larry

Reputation: 33

What you are fundamentally trying to do will not work...

Unfortunately exporting an RDS snapshot of Aurora MySQL to S3 and creating a new database cluster from the exported Snapshot will not be possible, this is because when you export a DB snapshot, Amazon RDS extracts data from the snapshot and stores it in an Amazon S3 bucket in your account. The data is stored in an Apache Parquet format that is compressed and consistent.

Please note that "restore from s3" and "export snapshot to s3" are serving for 2 different purposes.

Exporting a snapshot to S3 is a one way direction. The files in parquet format that have been exported to S3 can't be used to restore back to RDS. The purpose of it is to allow Amazon Athena or Amazon Redshift Spectrum to analyze data directly from s3.

On the other hand, restoring from s3 is for restoring a XtraBackup created on your local server which is for migrating data from an external MySQL database to an Amazon Aurora MySQL DB cluster

Exporting DB snapshot data to Amazon S3 https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ExportSnapshot.html

Migrating data from an external MySQL database to an Amazon Aurora MySQL DB cluster https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Migrating.ExtMySQL.html

Upvotes: 3

user488399
user488399

Reputation: 94

As Marcin commented, if your S3 export was encrypted, you have to grant KMS permission to the role proceeding the restore. I have added the role right in the KMS key policy under SID:"Allow use of the key". It is quite sad that AWS console does not display any clue about the missing encryption permissions. Currently, the AWS console forces you to specify the KMS key when exporting a MySQL snapshot to S3.

In my case, after granting the key permissions the console displayed a message

"Files from the specified Amazon S3 bucket are encrypted. Restoring from encrypted S3 object is not available"

.

S3 allows to disable the encryption only at the object level, so I walk through the 4 files of the snapshot and set the server-side encryption to Disabled. Then the restore process have started.

BTW I was not able to complete the restore with

"Incompatible-restore"

RDS instance status.

Upvotes: 0

Related Questions