com.amazonaws.services.s3.model.AmazonS3Exception: The XML you provided was not well-formed or did not validate against our published schema

Question

I am trying to delete content from an Amazon S3 bucket using deleteObjects method as below :

DeleteObjectsRequest deleteObjectsRequest = new DeleteObjectsRequest(s3BucketName);
        List<DeleteObjectsRequest.KeyVersion> keys = new ArrayList<>();
        //here s3 is the AmazonS3 object , bucketName is String Object & s3Filepath is also a string object.
        for (final S3ObjectSummary summary : S3Objects.withPrefix(s3, s3BucketName, s3FilePath)) {
            keys.add(new DeleteObjectsRequest.KeyVersion(summary.getKey()));
            System.out.println("Deleting the S3 object "+ summary);
        }
        deleteObjectsRequest.setKeys(keys);
        try {
            s3.deleteObjects(deleteObjectsRequest);
        } catch (Exception e) {
            System.out.print("Error when cleaning up S3-data: " + e);
        }

Reference has been taken from here : https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-multiple-objects.html

Weirdly , i'm getting below error message with catch block execution .

com.amazonaws.services.s3.model.AmazonS3Exception: The XML you provided was not well-formed or did not validate against our published schema (Service: Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID: someRequestID; S3 Extended Request ID: someExtendedRequestId=; Proxy: null)

Full StackTrace :

com.amazonaws.services.s3.model.AmazonS3Exception: The XML you provided was not well-formed or did not validate against our published schema (Service: Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID: SomeRequestId; S3 Extended Request ID: SomeExternalRequestId; Proxy: null)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819) ~[AWSJavaClientRuntime-1.11.x.jar:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403) ~[AWSJavaClientRuntime-1.11.x.jar:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372) ~[AWSJavaClientRuntime-1.11.x.jar:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ~[AWSJavaClientRuntime-1.11.x.jar:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) ~[AWSJavaClientRuntime-1.11.x.jar:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) ~[AWSJavaClientRuntime-1.11.x.jar:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) ~[AWSJavaClientRuntime-1.11.x.jar:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) ~[AWSJavaClientRuntime-1.11.x.jar:?]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) ~[AWSJavaClientRuntime-1.11.x.jar:?]
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) ~[AWSJavaClientRuntime-1.11.x.jar:?]
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) ~[AWSJavaClientRuntime-1.11.x.jar:?]
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437) ~[AWSS3JavaClient-1.11.x.jar:?]
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384) ~[AWSS3JavaClient-1.11.x.jar:?]
    at com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2327) ~[AWSS3JavaClient-1.11.x.jar:?]

Can someone please help here , what wrong i'm doing ?

Is this due to size limit of S3 bucket that it can only delete these much number of records . Because there are 10 files present in S3 & one file can have 1000 records .

Thanks !

Answer 1

I added the ETag in ExposeHeaders and it fixed the issue

Answer 2

S3 delete operation support up to 1000 object deletes, however, I can't see any mechanism in your code to safeguards keys more than 1000 passing to S3.

Can you print the length of the keys list? If it going beyond 1000, limit it and pass multiple delete invocations to S3.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html

Answer 3

Try moving to the latest Amazon S3 Java V2 API, not the old V1 API.

Here is the latest code example for this use case.

https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/s3/src/main/java/com/example/s3/DeleteObjects.java

If you are not familiar with how to work with the Amazon S3 Java V2 API, see:

Get started with the AWS SDK for Java 2.x

UPDATE

I just tested this V1 code and it worked perfectly:

import com.amazonaws.AmazonServiceException;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.DeleteObjectsRequest;

import java.util.Arrays;

/**
 * Delete multiple objects from an Amazon S3 bucket.
 *
 * This code expects that you have AWS credentials set up per:
 * http://docs.aws.amazon.com/java-sdk/latest/developer-guide/setup-credentials.html
 *
 * ++ Warning ++ This code will actually delete the objects that you specify!
 */
public class DeleteObjects {
    public static void main(String[] args) {
        final String USAGE = "\n" +
                "To run this example, supply the name of an S3 bucket and at least\n" +
                "one object name (key) to delete.\n" +
                "\n" +
                "Ex: DeleteObjects <bucketname> <objectname1> [objectname2, ...]\n";

     //   if (args.length < 2) {
      //      System.out.println(USAGE);
      ///      System.exit(1);
     //   }

        String bucket_name = "<bucketName>"; // args[0];
        String[] object_keys = {"book.pdf", "JUnit.pdf"}; // objects to delete

        System.out.println("Deleting objects from S3 bucket: " + bucket_name);
        for (String k : object_keys) {
            System.out.println(" * " + k);
        }

        final AmazonS3 s3 = AmazonS3ClientBuilder.standard()
                .withRegion(Regions.US_EAST_1)
                .build();


        try {
            DeleteObjectsRequest dor = new DeleteObjectsRequest(bucket_name)
                    .withKeys(object_keys);

            s3.deleteObjects(dor);
        } catch (AmazonServiceException e) {
            System.err.println(e.getErrorMessage());
            System.exit(1);
        }

        System.out.println("Done!");
    }
}