AWS S3 CopyObjectAsync fails with key does not exist, but get/put succeeds

Question

I fought with this for several hours and never found the solution. Here's the scenario:

var copyObjectRequest = new CopyObjectRequest
{
    SourceBucket  = s3Event.S3.Bucket.Name,
    SourceKey  = s3Event.S3.Object.Key,
    DestinationBucket  = OutputBucketName,
    DestinationKey  = s3Event.S3.Object.Key,
};

var deleteRequest = new DeleteObjectRequest
{
    BucketName = copyObjectRequest.SourceBucket,
    Key = copyObjectRequest.SourceKey,
};

await S3Client.CopyObjectAsync(copyObjectRequest);
await S3Client.DeleteObjectAsync(deleteRequest);

S3Client.CopyObjectAsync throws the error: "The specified key does not exist." (S3Client.DeleteObjectAsync is never reached.)

However, the following code works (for the same values):

var request = new GetObjectRequest
{
    BucketName = copyObjectRequest.SourceBucket,
    Key = copyObjectRequest.SourceKey,
};

var response = await S3Client.GetObjectAsync(request);
var tempPath = $"{Guid.NewGuid():D}";
await response.WriteResponseStreamToFileAsync(tempPath, false, CancellationToken.None);

var putRequest = new PutObjectRequest
{
    BucketName = copyObjectRequest.DestinationBucket,
    Key = copyObjectRequest.DestinationKey,
    FilePath = tempPath,
};

var putResponse = await S3Client.PutObjectAsync(putRequest);

if (putResponse.HttpStatusCode == HttpStatusCode.OK)
{
    var deleteRequest = new DeleteObjectRequest
    {
        BucketName = copyObjectRequest.SourceBucket,
        Key = copyObjectRequest.SourceKey,
    };

    await S3Client.DeleteObjectAsync(deleteRequest);
}

For brevity I removed almost all error checking, logging, etc. but if requested I'm happy to share the full function.

Note that this is running in a C# Lambda Function, using Core 2.0.

• I've ruled out security as the second set of calls requires the same permissions (I believe) as the CopyObject call does (please do correct me if I'm wrong).
• There's no doubt the object is at the bucket and key specified as the second set uses the exact same structure.
• The key doesn't exist in the destination bucket.
• Both the source and destination buckets have the same permissions.
• There are no special characters in the key (sample keys that I've tested are "/US/ID/Teton/EC2ClientDemo.zip" and "testkey").
• The files I'm testing with are tiny (that sample file is 30Kb).
• I've tried it with and without a CannedACL value in CopyObjectRequest (I don't think it should require one for my purposes, all the files it's moving around are private).
• I've validated that all buckets are in the same region (USWest2).

I can't figure out why CopyObjectAsync fails. I've tried digging down through the disassembled code for CopyObjectAsync, but it's called so indirectly I didn't get very far. At this point my best guess is that it's a bug in CopyObjectAsync.

Any suggestions would be appreciated, Thanks for reading!

Additional: I want to make it clear that this works from the regular AWSSDK library (either CopyObjectAsync or CopyObject), it only fails in the Core 2.0 async call CopyObjectAsync in the Lambda environment.

Answer 1

OK, so I figured it out and it is definitely a bug in the core 2.0 CopyObjectAsync(). Here's the scenario:

We are using keys that have slashes at the beginning, an example would be '/US/ID/Teton/EC2ClientDemo.zip'. When I turned on S3 logging (thank you @Michael-sqlbot) what I saw was this:

[13/Jul/2018:17:44:18 +0000] 34.221.84.59 arn:aws:sts::434371411556:assumed-role/LambdaFunctionCreation/TestFunction 489A5570C2E840AC REST.COPY.OBJECT_GET US/ID/Teton/EC2ClientDemo.zip - 404 NoSuchKey

As you can see the CopyObjectAsync() function stripped off the first slash before making the call. Get, Put, and Delete handle these particular keys just fine (and I tested this in the non-Core library and both the synchronous and asynchronous versions of CopyObjectAsync() handle the keys just fine as well).

What I had to do to fix it was the following:

var copyObjectRequest = new CopyObjectRequest
{
    SourceBucket  = s3Event.S3.Bucket.Name,
    SourceKey  = "/" + s3Event.S3.Object.Key,
    DestinationBucket  = OutputBucketName,
    DestinationKey  = "/" + s3Event.S3.Object.Key,
    CannedACL = S3CannedACL.BucketOwnerFullControl,
};

Note the added slashes on the SourceKey and DestinationKey? Without those the keys are mangled.

Here is the complete final code:

var copyObjectRequest = new CopyObjectRequest
{
    SourceBucket = s3Event.S3.Bucket.Name,
    SourceKey = s3Event.S3.Object.Key,
    DestinationBucket = OutputBucketName,
    DestinationKey = s3Event.S3.Object.Key,
    CannedACL = S3CannedACL.BucketOwnerFullControl,
};

try
{
    await s3Client.CopyObjectAsync(copyObjectRequest);
}
catch (AmazonS3Exception ase) when (ase.Message.Contains("key does not exist"))
{
    try
    {
        // If this failed due to Key not found, then fix up the request for the CopyObjectAsync bug in the Core 2.0 library and try again.
        var patchedCopyObjectRequest = new CopyObjectRequest()
                                       {
                                           SourceBucket  = copyObjectRequest.SourceBucket,
                                           SourceKey  = "/" + copyObjectRequest.SourceKey,
                                           DestinationBucket  = copyObjectRequest.DestinationBucket,
                                           DestinationKey  = "/" + copyObjectRequest.DestinationKey,
                                           CannedACL = copyObjectRequest.CannedACL,
                                       };

        await s3Client.CopyObjectAsync(patchedCopyObjectRequest);
    }
    catch (AmazonS3Exception)
    {
        // Rethrow the initial exception, since we don't want a confusing message to contain the modified keys.
        throw ase;
    }
}