CODEWITHSUNDEEP
aws-cloudformationaws-glue
botchniaque
botchniaque

Reputation: 5084

Cloudformation custom modules

Is there anything in Cloudformation similar to terraform modules, where you can create a parametrized template (group of resources, not CF template) and then use it in your CF template multiple times with different parameters?

I am seeing a lot of bolierplate YAML in my CF templates and I am looking for a way to refactor it. I have used terraform before and it provided that functionality.

Example:

I am creating many AWS Glue jobs, many of them differ with only 2 parameters, but each definition is 25 lines of code.

Resources:
  myGlueJob1:
    Type: AWS::Glue::Job
    Properties:
      ExecutionProperty:
        MaxConcurrentRuns: 1
      MaxRetries: 3
      Name: myGlueJob1
      Role: arn:aws:iam::xxxxxxx:role/XXXXXXXXXXXXXX
      Command:
        Name: glueetl
        ScriptLocation: XXXXXX
      DefaultArguments:
        "--ga_project_id": PARAM1-THAT-DIFFERS
        "--ga_view_id": PARAM2-THAT-DIFFERS
        "--ga_service_account_keyfile": gc.key-SAME_FOR_ALL_RESOURCES
        "--date": YESTERDAY-SAME_FOR_ALL_RESOURCES
        "--temp_gcs_bucket": "foobar-SAME_FOR_ALL_RESOURCES"
        "--output_path": "s3://some-s3-path-SAME_FOR_ALL_RESOURCES"
        "--job-language": scala
        "--class": GlueApp
        "--TempDir": "s3://some-other-s3-path-SAME_FOR_ALL_RESOURCES"
        "--extra-files": "s3://some-s3-path-SAME_FOR_ALL_RESOURCES"
        "--extra-jars": "s3://looooooooooooooooooooooooots-of-s3-paths-SAME_FOR_ALL_RESOURCES"
  myGlueJob2:
    Type: AWS::Glue::Job
    Properties:
      ExecutionProperty:
        MaxConcurrentRuns: 1
      MaxRetries: 3
      Name: myGlueJob2
      Role: arn:aws:iam::xxxxxxx:role/XXXXXXXXXXXXXX
      Command:
        Name: glueetl
        ScriptLocation: XXXXXX
      DefaultArguments:
        "--ga_project_id": PARAM1-THAT-DIFFERS
        "--ga_view_id": PARAM2-THAT-DIFFERS
        "--ga_service_account_keyfile": gc.key-SAME_FOR_ALL_RESOURCES
        "--date": YESTERDAY-SAME_FOR_ALL_RESOURCES
        "--temp_gcs_bucket": "foobar-SAME_FOR_ALL_RESOURCES"
        "--output_path": "s3://some-s3-path-SAME_FOR_ALL_RESOURCES"
        "--job-language": scala
        "--class": GlueApp
        "--TempDir": "s3://some-other-s3-path-SAME_FOR_ALL_RESOURCES"
        "--extra-files": "s3://some-s3-path-SAME_FOR_ALL_RESOURCES"
        "--extra-jars": "s3://looooooooooooooooooooooooots-of-s3-paths-SAME_FOR_ALL_RESOURCES"

I can imagine a solution which looks something like this:

Module:
    Type: Me::MyGlueJob
    Resouces:
        Type: AWS::Glue::Job
        Properties:
          ExecutionProperty:
            MaxConcurrentRuns: 1
          MaxRetries: 3
          Name: myGlueJob2
          Role: arn:aws:iam::xxxxxxx:role/XXXXXXXXXXXXXX
          Command:
            Name: glueetl
            ScriptLocation: XXXXXX
          DefaultArguments:
            "--ga_project_id": {{ MY_PARAM1 }}
            "--ga_view_id": {{ MY_PARAM2 }}
            "--ga_service_account_keyfile": gc.key-SAME_FOR_ALL_RESOURCES
            "--date": YESTERDAY-SAME_FOR_ALL_RESOURCES
            "--temp_gcs_bucket": "foobar-SAME_FOR_ALL_RESOURCES"
            "--output_path": "s3://some-s3-path-SAME_FOR_ALL_RESOURCES"
            "--job-language": scala
            "--class": GlueApp
            "--TempDir": "s3://some-other-s3-path-SAME_FOR_ALL_RESOURCES"
            "--extra-files": "s3://some-s3-path-SAME_FOR_ALL_RESOURCES"
            "--extra-jars": "s3://looooooooooooooooooooooooots-of-s3-paths-SAME_FOR_ALL_RESOURCES"
Resources:
  myGlueJob1:
    Type: Me::MyGlueJob
    Properties:
        MY_PARAM1: value-for-job1
        MY_PARAM2: value-for-job1
  myGlueJob2:
    Type: Me::MyGlueJob
    Properties:
        MY_PARAM1: value-for-job2
        MY_PARAM2: value-for-job2

Any hint on best practices would be very appreciated.

Upvotes: 0

Views: 1018

Answers (1)

Yuriy Bondaruk
Yuriy Bondaruk

Reputation: 4750

You should be able to use jinja2 templates to generate actual CloudFormation templates.

In your case it should look like this:

{% set job_params = [
  ["value-for-job1", "value-for-job1"],
  ["value-for-job2", "value-for-job2"]
] %}

Resources:
{% for params in job_params %}
  myGlueJob{{loop.index}}:
    Type: AWS::Glue::Job
    Properties:
      ExecutionProperty:
        MaxConcurrentRuns: 1
      MaxRetries: 3
      Name: myGlueJob1
      Role: arn:aws:iam::xxxxxxx:role/XXXXXXXXXXXXXX
      Command:
        Name: glueetl
        ScriptLocation: XXXXXX
      DefaultArguments:
        "--ga_project_id": "{{params[0]}}"
        "--ga_view_id": "{{params[1]}}"
        "--ga_service_account_keyfile": gc.key-SAME_FOR_ALL_RESOURCES
        "--date": YESTERDAY-SAME_FOR_ALL_RESOURCES
        "--temp_gcs_bucket": "foobar-SAME_FOR_ALL_RESOURCES"
        "--output_path": "s3://some-s3-path-SAME_FOR_ALL_RESOURCES"
        "--job-language": scala
        "--class": GlueApp
        "--TempDir": "s3://some-other-s3-path-SAME_FOR_ALL_RESOURCES"
        "--extra-files": "s3://some-s3-path-SAME_FOR_ALL_RESOURCES"
        "--extra-jars": "s3://looooooooooooooooooooooooots-of-s3-paths-SAME_FOR_ALL_RESOURCES"
{% endfor %}

Upvotes: 1

Related Questions