CODEWITHSUNDEEP
amazon-web-servicesaws-glue
Omegaspard
Omegaspard

Reputation: 1980

Aws Glue job, update the script location

I struggle to find a way to update the script location of my aws glue job due with the aws cli.

https://docs.aws.amazon.com/cli/latest/reference/glue/update-job.html

I tried :

aws glue update-job --job-name "${JOB}-${ENV}" --job-update Command={Name="glueetl", ScriptLocation="s3://aws-glue-jobs-${ENV}/artifacts/${JOB}/${JOB}.py"}

And it return me the error :

Unknown options: ScriptLocation=s3://aws-glue-jobs-staging/artifacts/dummy_job/dummy_job.py}

I don't see where my error is. This is specified in the documentation.

Upvotes: 0

Views: 3468

Answers (1)

Ilya Kisil
Ilya Kisil

Reputation: 2668

Your problem stems from the way bash (or awscli) parses arguments for a given command, subcommand and parameters, i.e. it is sensitive to whitespaces and equal signs (=). The following should work just fine:

aws glue update-job --job-name="${JOB}-${ENV}" --job-update="Role=arn:aws:iam::__FILL_ME__:role/__FILL_ME__, Command={Name=glueetl, ScriptLocation=s3://aws-glue-jobs-${ENV}/artifacts/${JOB}/${JOB}.py}"

aws glue update-job --job-name "${JOB}-${ENV}" --job-update "Role=arn:aws:iam::__FILL_ME__:role/__FILL_ME__, Command={Name=glueetl, ScriptLocation=s3://aws-glue-jobs-${ENV}/artifacts/${JOB}/${JOB}.py}"

aws glue update-job \
    --job-name "${JOB}-${ENV}" \
    --job-update "Role=arn:aws:iam::__FILL_ME__:role/__FILL_ME__, Command={Name=glueetl, ScriptLocation=aws-glue-jobs-${ENV}/artifacts/${JOB}/${JOB}.py}"

aws glue update-job \
    --job-name="${JOB}-${ENV}" \
    --job-update="Role=arn:aws:iam::__FILL_ME__:role/__FILL_ME__, Command={Name=glueetl, ScriptLocation=aws-glue-jobs-${ENV}/artifacts/${JOB}/${JOB}.py}"

Note: It is required to specify Role=arn:aws:iam::__FILL_ME__, although your original snippet didn't contain one.

Side note on argument parsing

In the command you provided:

  • glue: name of the command
  • update-job: name of the subcommand
  • Everything after are key-value parameters (options) where key and value should be separated either by whitespace or equal sign (=)

Since there is a whitespace before ScriptLocation, parameters are parsed something like:

# First key-value pair
--job-name "${JOB}-${ENV}"

# Second key-value pair
--job-update Command={Name="glueetl",

# Third key-value pair
ScriptLocation="s3://aws-glue-jobs-${ENV}/artifacts/${JOB}/${JOB}.py"}

Or something like that. In general, preserving whitespaces can be accomplished by surrounding everything with either double or single quotation marks. Since you use environmental variables, you have to use double quotes, however, this would require to remove " for the values of Name and ScriptLocation. Don't worry about AWS docs asking specifically for string type. Bash doesn't have types in the same way as for example Python, so everything is interpreted as a string by default anyway.

Upvotes: 2

Related Questions