CODEWITHSUNDEEP
amazon-s3boto3
Dawny33
Dawny33

Reputation: 11081

S3 giving me NoSuchKey error even when the key exists

This is my boto3 command for getting the object with a specific key from an S3 bucket:

resp = s3client.get_object(Bucket='<>-<>', Key='MzA1MjY1NzkzX2QudHh0')

It gives the following error:

botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

I have checked in the bucket, and the key actually exists

enter image description here

Did I miss something or did I do something wrong here?

Upvotes: 35

Views: 164180

Answers (7)

Anamsken
Anamsken

Reputation: 51

This issue got me stuck for over 3 hours|

  1. First I updated the IAM to allow getobjet permission
  2. Second I added a waiter before the copy command

My code looks somewhat like this

    copy_source = {'Bucket': source_bucket, 'Key': object_key}
        
        waiter = s3_client.gt_waiter('object_exists')
        waiter.wait(Bucket=source_bucket, Key=object_key)

        s3_client.copy_object(Bucket=target_bucket, Key=object_key, CopySource=
copy_source)

Upvotes: 0

element11
element11

Reputation: 4475

In my case, the object key was coming in from a 3rd party API as /path/to/object.ext. The NoSuchKey error was due to the leading slash on the key param.

Other commands, such as CopyObjectCommand have a single CopySource param, so bucket-name/path/to/object.ext works for that, but with GetObject, bucket and key are separate params.

Removing the leading / clears up the error in this case

Upvotes: 2

Nagev
Nagev

Reputation: 13237

A generic answer that may useful to those who are thinking about file paths and may be new to the AWS S3 terminology. Not getting the "name" and "key" right will often lead to an exception with the message An error occurred (NoSuchKey)... as posted in this question.

Let's say that you have a JPEG file stored in some "path" in a bucket. Navigating to this object on the AWS console shows the S3 URI to be:

s3://my-bucket/some/very/long/path/my-image.jpeg

You could read the my-image.jpeg object in Python with this basic example:

import boto3

s3client = boto3.client('s3', region_name='us-east-1')

bucket_name = 'my-bucket'
object_key = 'some/very/long/path/my-image.jpeg'

try:
    s3obj = s3client.get_object(Bucket=bucket_name, Key=object_key)
except Exception as e:
    print(f"Error reading key {object_key} from bucket {bucket_name}: {e}")
else:
   print(f"Got object: {s3obj}")

Upvotes: 8

Tushar Kale
Tushar Kale

Reputation: 169

from urllib.parse import unquote
key = "MzA1MjY1NzkzX2QudHh0%0A"

key = unquote(key)

This will remove all special characters from any key.

Upvotes: 0

Cesar Bielich
Cesar Bielich

Reputation: 4945

Another possible issue that I came across that was causing a line separator in my url for an object was that in one of AWS documentation for getting the key of an object shows this code as an example to get the key of an object.

foreach ($results as $result) {
    foreach ($result['Contents'] as $object) {
        echo $object['Key'] . PHP_EOL;
    }
}

The problem is the PHP_EOL at the end. Just remove it and the line separator goes away.

$object['Key'] . PHP_EOL; --> $object['Key'];

Upvotes: 0

cookiedough
cookiedough

Reputation: 3832

Since you know the key that you have is definitely in the name of the file you are looking for, I recommend using a filter to get objects with names with your key as their prefix. 

s3 = boto3.resource('s3')
bucket = s3.Bucket('cypher-secondarybucket')
for obj in bucket.objects.filter(Prefix='MzA1MjY1NzkzX2QudHh0'):
    print obj.key

When you run this code, you will get the key names of all the files that start with your key. This will help you find out what your file is exactly called on S3.

Upvotes: 8

Chris Pollard
Chris Pollard

Reputation: 1780

You have a %0A at the end of your URL; that's a line separator.

Upvotes: 12

Related Questions