CODEWITHSUNDEEP
javaamazon-web-servicesamazon-s3aws-sdk
in His Steps
in His Steps

Reputation: 3245

How to check if a specified key exists in a given S3 bucket using Java

I would like to check if a key exists in a given bucket using Java. I looked at the API but there aren't any methods that are useful. I tried to use getObject but it threw an exception.

Upvotes: 141

Views: 240571

Answers (17)

malana
malana

Reputation: 5230

There's now a doesObjectExist method in the official Java API.

Enjoy!

EDIT: This works only in AWS SDK for Java 1.x.

Upvotes: 355

E.T
E.T

Reputation: 651

Update for the AWS SDK for Java 2.x

The right way to do it in SDK V2, without the overload of actually getting the object, is to use S3Client.headObject(). Officially backed by AWS Change Log in the [4.1.1. S3 Operation Migration][2] section.

Example code:

public boolean exists(String bucket, String key) {
    try {
        HeadObjectResponse headResponse = client
                .headObject(HeadObjectRequest.builder().bucket(bucket).key(key).build());
        return true;
    } catch (NoSuchKeyException e) {
        return false;
    }
}

More info about the HeadObject operation can be found in the official documentation: HeadObject.

Upvotes: 65

Mark Day
Mark Day

Reputation: 17

Maybe this will work ?

    /**
 * Exist object
 *
 * @param bucketName - Bucket name
 * @param key        - Object key
 * @return Mono<Boolean>
 */
public Mono<Boolean> existObject(String bucketName, String key) {
    return Mono.fromFuture(client.headObject(HeadObjectRequest.builder().key(key).bucket(bucketName).build()))
            .flatMap(result -> Mono.just(true))
            .onErrorResume(error -> Mono.just(false));
}

Upvotes: 0

koolhead17
koolhead17

Reputation: 1964

Alternatively you can use Minio-Java client library, its Open Source and compatible with AWS S3 API.

You can use Minio-Java StatObject.java examples for the same.

import io.minio.MinioClient;
import io.minio.errors.MinioException;

import java.io.InputStream;
import java.io.IOException;
import java.security.NoSuchAlgorithmException;
import java.security.InvalidKeyException;

import org.xmlpull.v1.XmlPullParserException;


public class GetObject {
  public static void main(String[] args)
    throws NoSuchAlgorithmException, IOException, InvalidKeyException, XmlPullParserException, MinioException {
    // Note: YOUR-ACCESSKEYID, YOUR-SECRETACCESSKEY and my-bucketname are
    // dummy values, please replace them with original values.
    // Set s3 endpoint, region is calculated automatically
    MinioClient s3Client = new MinioClient("https://s3.amazonaws.com", "YOUR-ACCESSKEYID", "YOUR-SECRETACCESSKEY");
    InputStream stream = s3Client.getObject("my-bucketname", "my-objectname");

    byte[] buf = new byte[16384];
    int bytesRead;
    while ((bytesRead = stream.read(buf, 0, buf.length)) >= 0) {
      System.out.println(new String(buf, 0, bytesRead));
    }

    stream.close();
  }
}

I hope it helps.

Disclaimer : I work for Minio

Upvotes: -1

Navigatron
Navigatron

Reputation: 2115

As others have mentioned, for the AWS S3 Java SDK 2.10+ you can use the HeadObjectRequest object to check if there is a file in your S3 bucket. This will act like a GET request without actually getting the file.

Example code since others haven't actually added any code above:

public boolean existsOnS3 () throws Exception {
    try {
       S3Client s3Client = S3Client.builder ().credentialsProvider (...).build ();
       HeadObjectRequest headObjectRequest = HeadObjectRequest.builder ().bucket ("my-bucket").key ("key/to/file/house.pdf").build ();
       HeadObjectResponse headObjectResponse = s3Client.headObject (headObjectRequest);
       return headObjectResponse.sdkHttpResponse ().isSuccessful ();    
   }
   catch (NoSuchKeyException e) {
      //Log exception for debugging
      return false;
   }
}

Upvotes: 4

Aakash Sharma
Aakash Sharma

Reputation: 39

I also faced this problem when I used

String BaseFolder = "3patti_Logs"; 
S3Object object = s3client.getObject(bucketName, BaseFolder);

I got error key not found

When I hit and try

String BaseFolder = "3patti_Logs"; 
S3Object object = s3client.getObject(bucketName, BaseFolder+"/");

it worked , this code is working with 1.9 jar otherwise update to 1.11 and use doesObjectExist as said above

Upvotes: 1

Paul Draper
Paul Draper

Reputation: 83393

In Amazon Java SDK 1.10+, you can use getStatusCode() to get the status code of the HTTP response, which will be 404 if the object does not exist.

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.AmazonS3Exception;
import org.apache.http.HttpStatus;

try {
    AmazonS3 s3 = new AmazonS3Client();
    ObjectMetadata object = s3.getObjectMetadata("my-bucket", "my-client");
} catch (AmazonS3Exception e) {
    if (e.getStatusCode() == HttpStatus.SC_NOT_FOUND) {
        // bucket/key does not exist 
    } else {
        throw e;
    }
}

getObjectMetadata() consumes fewer resources, and the response doesn't need to be closed like getObject().

In previous versions, you can use getErrorCode() and check for the appropriate string (depends on the version).

Upvotes: 19

AlikElzin-kilaka
AlikElzin-kilaka

Reputation: 36051

Update:

It seems there's a new API to check just that. See another answer in this page: https://stackoverflow.com/a/36653034/435605

Original post:

Use errorCode.equals("NoSuchKey")

try {
    AmazonS3 s3 = new AmazonS3Client(new ClasspathPropertiesFileCredentialsProvider());
    String bucketName = getBucketName();
    s3.createBucket(bucketName);
    S3Object object = s3.getObject(bucketName, getKey());
} catch (AmazonServiceException e) {
    String errorCode = e.getErrorCode();
    if (!errorCode.equals("NoSuchKey")) {
        throw e;
    }
    Logger.getLogger(getClass()).debug("No such key!!!", e);
}

Note about the exception: I know exceptions should not be used for flow control. The problem is that Amazon didn't provide any api to check this flow - just documentation about the exception.

Upvotes: 68

Bampfer
Bampfer

Reputation: 2220

The other answers are for AWS SDK v1. Here is a method for AWS SDK v2 (currently 2.3.9).

Note that getObjectMetadata and doesObjectExist methods are not currently in the v2 SDK! So those are no longer options. We are forced to use either getObject or listObjects.

listObjects calls are currently 12.5 times more expensive to make than getObject. But AWS also charges for any data downloaded, which raises the price of getObject if the file exists. As long as the file is very unlikely to exist (for example, you have generated a new UUID key randomly and just need to double-check that it isn't taken) then calling getObject is significantly cheaper by my calculation.

Just to be on the safe side though, I added a range() specification to ask AWS to only send a few bytes of the file. As far as I know the SDK will always respect this and not charge you for downloading the whole file. But I haven't verified that so rely on that behavior at your own risk! (Also, I'm not sure what how range behaves if the S3 object is 0 bytes long.)

    private boolean sanityCheckNewS3Key(String bucket, String key) {

        ResponseInputStream<GetObjectResponse> resp = null;
        try {
            resp = s3client.getObject(GetObjectRequest.builder()
                .bucket(bucket)
                .key(key)
                .range("bytes=0-3")
                .build());
        }
        catch (NoSuchKeyException e) {
            return false;
        }
        catch (AwsServiceException se) {
            throw se;
        }
        finally {
            if (resp != null) {
                try {
                    resp.close();
                } catch (IOException e) {
                    log.warn("Exception while attempting to close S3 input stream", e);
                }
            }
        }
        return true;
    }
}

Note: this code assumes s3Client and log are declared and initialized elsewhere. Method returns a boolean, but can throw exceptions.

Upvotes: 3

Kaustuv
Kaustuv

Reputation: 788

Using Object isting. Java function to check if specified key exist in AWS S3.

boolean isExist(String key)
    {
        ObjectListing objects = amazonS3.listObjects(new ListObjectsRequest().withBucketName(bucketName).withPrefix(key));

        for (S3ObjectSummary objectSummary : objects.getObjectSummaries())
        {
            if (objectSummary.getKey().equals(key))
            {
                return true;
            }

        }
        return false;
    }

Upvotes: 2

roee
roee

Reputation: 908

Break your path into bucket and object. Testing the bucket using the method doesBucketExist, Testing the object using the size of the listing (0 in case not exist). So this code will do:

String bucket = ...;
String objectInBucket = ...;
AmazonS3 s3 = new AmazonS3Client(...);
return s3.doesBucketExist(bucket) 
       && !s3.listObjects(bucket, objectInBucket).getObjectSummaries().isEmpty();

Upvotes: 3

Dhwaneel
Dhwaneel

Reputation: 551

There is an easy way to do it using jetS3t API's isObjectInBucket() method.

Sample code:

ProviderCredentials awsCredentials = new AWSCredentials(
                awsaccessKey,
                awsSecretAcessKey);

        // REST implementation of S3Service
        RestS3Service restService = new RestS3Service(awsCredentials);

        // check whether file exists in bucket
        if (restService.isObjectInBucket(bucket, objectKey)) {

            //your logic

        }

Upvotes: 1

Rich Remer
Rich Remer

Reputation: 2613

For PHP (I know the question is Java, but Google brought me here), you can use stream wrappers and file_exists

$bucket = "MyBucket";
$key = "MyKey";
$s3 = Aws\S3\S3Client->factory([...]);
$s3->registerStreamWrapper();
$keyExists = file_exists("s3://$bucket/$key");

Upvotes: 5

c0mrade
c0mrade

Reputation: 1390

This java code checks if the key (file) exists in s3 bucket.

public static boolean isExistS3(String accessKey, String secretKey, String bucketName, String file) {

    // Amazon-s3 credentials
    AWSCredentials myCredentials = new BasicAWSCredentials(accessKey, secretKey); 
    AmazonS3Client s3Client = new AmazonS3Client(myCredentials); 

    ObjectListing objects = s3Client.listObjects(new ListObjectsRequest().withBucketName(bucketName).withPrefix(file));

    for (S3ObjectSummary objectSummary: objects.getObjectSummaries()) {
        if (objectSummary.getKey().equals(file)) {
            return true;
        }
    }
    return false;
}

Upvotes: 4

user34402
user34402

Reputation: 147

Use ListObjectsRequest setting Prefix as your key.

.NET code:

 public bool Exists(string key)
    {

        using (Amazon.S3.AmazonS3Client client = (Amazon.S3.AmazonS3Client)Amazon.AWSClientFactory.CreateAmazonS3Client(m_accessKey, m_accessSecret))
        {
            ListObjectsRequest request = new ListObjectsRequest();
            request.BucketName = m_bucketName;
            request.Prefix = key;
            using (ListObjectsResponse response = client.ListObjects(request))
            {

                foreach (S3Object o in response.S3Objects)
                {
                    if( o.Key == key )
                        return true;
                }
                return false;
            }
        }
    }.

Upvotes: 5

user979051
user979051

Reputation: 1267

Using the AWS SDK use the getObjectMetadata method. The method will throw an AmazonServiceException if the key doesn't exist.

private AmazonS3 s3;
...
public boolean exists(String path, String name) {
    try {
        s3.getObjectMetadata(bucket, getS3Path(path) + name); 
    } catch(AmazonServiceException e) {
        return false;
    }
    return true;
}

Upvotes: 26

sethu
sethu

Reputation: 8431

Use the jets3t library. Its a lot more easier and robust than the AWS sdk. Using this library you can call, s3service.getObjectDetails(). This will check and retrieve only the details of the object (not the contents) of the object. It will throw a 404 if the object is missing. So you can catch that exception and deal with it in your app.

But in order for this to work, you will need to have ListBucket access for the user on that bucket. Just GetObject access will not work. The reason being, Amazon will prevent you from checking for the presence of the key if you dont have ListBucket access. Just knowing whether a key is present or not, will also suffice for malicious users in some cases. Hence unless they have ListBucket access they will not be able to do so.

Upvotes: 0

Related Questions