CODEWITHSUNDEEP
amazon-web-services
rodney ellis
rodney ellis

Reputation: 31

AWS CLI create batch job

I have a bash script that creates a manifest.csv file example below, and then I use that to create an S3 batch job, I ran it using the console and it works so roles and permissions are correct, any help will be appreciated.

test-bucket-batch,Test/testing1.json \
test-bucket-batch,Test/testing2.json \
test-bucket-batch,Test/testing3.json \
test-bucket-batch,Test/testing4.json 

aws s3control create-job \
    --account-id $ACCOUNT_ID \
    --region $REGION \
    --confirmation-required \
    --client-request-token $(uuidgen) \
    --operation '{"S3PutObjectCopy":{"TargetResource":"arn:aws:s3:::'$BUCKET_NAME'/object_restore/"}}' \
    --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},"Location":{"ObjectArn":"arn:aws:s3:::'$BUCKET_NAME'/object_restore/manifest.csv","ETag":'$ETAG'}}' \
    --report '{"Bucket":"arn:aws:s3:::'$BUCKET_NAME'/object_restore/","Prefix":"final-reports", "Format":"Report_CSV_20180820","Enabled": true,"ReportScope":"AllTasks"}' \
    --description 'S3 Copy Job' \
    --priority 42 \
    --role-arn $ROLE_ARN

ERROR: An error occurred (InvalidRequest) when calling the CreateJob operation: Request invalid

Upvotes: 0

Views: 1296

Answers (1)

rodney ellis
rodney ellis

Reputation: 31

The issue was related to the prefix

aws s3control create-job \
--account-id $ACCOUNT_ID \
--region $REGION \
--confirmation-required \
--client-request-token $(uuidgen) \
--operation '{"S3PutObjectCopy":{"TargetResource":"arn:aws:s3:::'$BUCKET_NAME'","TargetKeyPrefix":"/object_restore/"}}' \
--manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},"Location":{"ObjectArn":"arn:aws:s3:::'$BUCKET_NAME'/object_restore/manifest.csv","ETag":'$ETAG'}}' \
--report '{"Bucket":"arn:aws:s3:::'$BUCKET_NAME'","Prefix":"/object_restore/final-reports", "Format":"Report_CSV_20180820","Enabled": true,"ReportScope":"AllTasks"}' \
--description 'S3 Copy Job' \
--priority 42 \
--role-arn $ROLE_ARN

Note the fields changed where:

  1. operation where TargetKeyPrefix was added: "TargetKeyPrefix":"/object_restore/"
  2. report where Prefix was changed to: "Prefix":"/object_restore/final-reports"

Upvotes: 3

Related Questions