CODEWITHSUNDEEP
yamlaws-cloudformationaws-glue
malana
malana

Reputation: 5220

How do I pass arguments to a Glue job in CloudFormation YAML?

You can pass arguments to an AWS Glue job via the --arguments parameter (see here).

The CloudFormation documentation says DefaultArguments are "UTF-8 string–to–UTF-8 string key-value pairs" and that their type is "JSON object". Since YAML is a super set of JSON, I was expecting to be able to pass arguments like this in a (YAML) CloudFormation template:

DefaultArguments:
  "--arguments": {"--test_argument": "foo"}

However, it raises this error during CloudFormation deployment:

Property validation failure: [Value of property {/DefaultArguments/--arguments=} does not match type {String}]

How do I specify the values correctly?

Upvotes: 4

Views: 8028

Answers (2)

malana
malana

Reputation: 5220

The correct way of passing multiple parameters is

DefaultArguments:
  "--argument1": value1
  "--argument2": value2

and then accessing them in the job (e.g. in Python) like this:

from awsglue.utils import getResolvedOptions
args = getResolvedOptions(sys.argv, ['argument1', 'argument2'])
print args['argument1']
print args['argument2']

What confused me was that for passing parameters with the AWS CLI, you use an explicit --arguments='--argument1="value1"' structure but in CloudFormation, you specify arguments one by one.

Upvotes: 10

Anthon
Anthon

Reputation: 76578

The value for the key --arguments needs to be a string, but you actually give it a mapping (or in JSON-speak an object), because it starts witha {. You should quote the value, and since you have double quotes in the value, you best do that with single quotes:

DefaultArguments:
  "--arguments": '{"--test_argument": "foo"}'

(any existing single quotes in the value you would need to escape by putting two single quotes)

If your JSON is more complex it can be benificial to use folded-style scalars. Within those the { has no special meaning either and (single) newlines followed by spaces are replaced by a single space. So the following loads to same data as the solution above:

DefaultArguments:
  "--arguments": >
    {"--test_argument": 
        "foo"}

Of course with YAML (1.2) being a superset of JSON, glue could easily assume that a value is already parsed if it is not a string, but it doesn't seem to be that smart and always expects the JSON in string form.

Upvotes: 1

Related Questions