CODEWITHSUNDEEP
pythonamazon-web-servicesaws-lambdaboto3aws-glue
Aatisha
Aatisha

Reputation: 3

AWS Lambda error: not all arguments converted during string formatting

I have a python lambda code which iterates over a loop and starts 17 instances of a reusable glue job by passing run time arguments. I am using boto3 function which is start_job_run() to start the job. Below is the snippet:

    job_keys = {}
    for job_key_name in jobs:
        try:
            job_keys[job_key_name] = glue_client.start_job_run(JobName = glueJobName, Arguments = {'--Path': CONFIG_PATH, '--JobKeyName': job_key_name}) <--- Error is happening here
            logger.info(f'## STARTED GLUE JOB: {glueJobName}-{job_key_name}')
            logger.info(f'## GLUE JOB RUN ID: {job_keys[job_key_name]["JobRunId"]}')
            logger.info(f'## SOURCE: {source}')
        except Exception as e:
            logger.error('ERROR DETAILS', e)
            logger.error(job_key_name)
            pass

The code runs fine and starts 17 executions of the glue job. But there are few intermittent instances when the lambda fails with the error not all arguments converted during string formatting. When this happens the particular job instance does not start.

Note: The max concurrency of the glue job is set to 17 and no. of workers is 2.

Upvotes: 0

Views: 1310

Answers (1)

lynkfox
lynkfox

Reputation: 2400

Your error is actually occurring in the Exception itself, but the error is masking another.

logger.error('ERROR DETAILS', e)

when receiving a botocore.ClientError exception will fail at this point with the mentioned error as logger.error has no idea how to process a boto3 ClientError object into a string (there are a lot of additional items on the ClientException object)

If you add the following:

from botocore.exceptions import ClientError
... [ your code ]
except ClientError as e:
    ...[handle the client error specifically]
except Exception as e:
    ...[the rest of your exception handling]

it should allow you to start to see the Client Error issue that is occurring - as it is intermittent, my guess would be that you are running up against glue limits for too many concurrent executions

Upvotes: 1

Related Questions