CODEWITHSUNDEEP
pythonamazon-s3aws-lambdaboto3
lakemap
lakemap

Reputation: 121

Can't create S3 batch job through boto3 - gets request invalid

I am trying to create an S3 Batch (not AWS Batch, this is S3 Batch operation) job via boto3 using S3Control, but I get an "invalid request" response. I tried it through AWS S3 batch operation through the console which worked but now I am trying to do it through boto3 to create the batch job. Attached is the code and error message.

import json
import boto3
import time
import datetime
import os
s3ControlClient = boto3.client('s3control')
def lambda_handler(event, context):
    accountId = boto3.client('sts').get_caller_identity().get('Account')
    print(accountId)
    response = s3ControlClient.create_job(
        AccountId = accountId,
        ConfirmationRequired = False,
        Operation = {
            'S3PutObjectRetention': {
                'BypassGovernanceRetention': False,
                'Retention': {
                    'RetainUntilDate'   : datetime.datetime(2050, 06, 03),
                    'Mode'              : 'GOVERNANCE'
                }
            }
        },
        Report={
            'Bucket'        : 'arn:aws:s3:::test-s3-inventory',
            'Format'        : 'Report_CSV_20180820',
            'Enabled'       : True,
            #'Prefix'        : time.strftime('%Y%m%d'),
            'Prefix'        : 'report',
            'ReportScope'   : 'AllTasks'
        },
        Manifest={
            'Spec': {
                'Format': 'Report_CSV_20180820',
                'Fields': [
                    'Bucket', 'Key', 'VersionId', 'TaskStatus', 'ErrorCode', 'HTTPStatusCode', 'ResultMessage'
                ]
            },
            'Location': {
                'ObjectArn'         : 'https://test-s3-inventory.s3.amazonaws.com/job-34455-4eb5-829d-7eedrrr8564/manifest.json',
                'ETag'              : 'f4a7c0618aaed7777a5be40c266abe1f'
              }
        },
        Description = time.strftime('%Y-%m-%d')+' - Apply Retention and Legal Hold',
        Priority    = 10,
        RoleArn     = 'arn:aws:iam::277696433388194:role/s3BatchRole',
        Tags        = [
            {'Key': LOB', 'Value': 'Test'},
            {'Key': 'project',       'Value': ‘test project’}
        ]
    )

Error:
    Response:
    {
      "errorMessage": "An error occurred (InvalidRequest) when calling the CreateJob operation: Request invalid",
      "errorType": "ClientError",
      "stackTrace": [
        "  File \"/var/task/lambda_function.py\", line 13, in lambda_handler\n    response = s3ControlClient.create_job(\n",
        "  File \"/opt/python/botocore/client.py\", line 316, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n",
        "  File \"/opt/python/botocore/client.py\", line 635, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n"
      ]
    }

Upvotes: 5

Views: 4791

Answers (2)

Saurav Agrawal
Saurav Agrawal

Reputation: 101

Additionally, ClientRequestToken is missing. According to docs,

ClientRequestToken (string) -- [REQUIRED]

An idempotency token to ensure that you don't accidentally submit the same request twice. You can use any string up to the maximum length.

This field is autopopulated if not provided.

ClientRequestToken can be added in the following way -

import uuid
...

def lambda_handler(event, context):
    ...
    RequestToken = str(uuid.uuid4())
    ...
    response = clientControl.create_job(
      ...
      ClientRequestToken=RequestToken,
      ...
    )
    ....

Also you need to specify the region when you are creating the boto3 client as best practice.

s3ControlClient=boto3.client('s3control', region_name=[Region_Name])

I have documented my syntax of using create job in S3 Control Create Job Invalid Request Error

Upvotes: 0

Michael Calcagno
Michael Calcagno

Reputation: 51

I think you have two small errors in your code:

  1. As @Limsanity82 noted, your object ARN should be in the following format: 
    arn:aws:s3:::test-s3-inventory/job-34455-4eb5-829d-7eedrrr8564/manifest.json
  2. In your manifest spec, the formatting value (Report_CSV_20180820) is wrong; instead you want: 
    S3BatchOperations_CSV_20180820

Upvotes: 5

Related Questions