merge lists of dictionaries in terraform v0.12

Question

I would like to do the following using terraform:

I have 2 JSONs:

1.json:

[
    {
        "description": "description1",
        "url":         "url1",
        "data":        "data1"
    },
    {
        "description": "description2",
        "url":         "url2",
        "data":        "data2",
        "action":        "action2"
    },
    {
        "description": "description3",
        "url":         "url3",
        "data":        "data3"
    }
]

2.json:

[
    {  
        "description": "description1",
        "url":         "url1",
        "data":        "data1"
    },
    {
        "description": "description2_new",
        "url":         "url2",
        "data":        "data2_new"
    },
    {
        "description": "description4",
        "url":         "url4",
        "data":        "data4"
    }
]

and I want to merge them into one. Dictionaries from the second JSON should override dictionaries from the first one if url key is the same. I.e. combined JSON should look like:

[
    {
        "description": "description1",
        "url":         "url1",
        "data":        "data1"
    },
    {
        "description": "description2_new",
        "url":         "url2",
        "data":        "data2_new"
    },
    {
        "description": "description3",
        "url":         "url3",
        "data":        "data3"
    },
    {
        "description": "description4",
        "url":         "url4",
        "data":        "data4"
    }
]

Using python I can easily do it:

import json
with open('1.json') as f:
  json1 = json.load(f)
with open('2.json') as f:
  json2 = json.load(f)


def list_to_dict(json_list):
    res_dict = {}
    for d in json_list:
        res_dict[d['url']] = d
    return res_dict

def merge_json(json1, json2):
    j1 = list_to_dict(json1)
    j2 = list_to_dict(json2)
    j1.update(j2)
    res_list = []
    for key in j1.keys():
        res_list.append(j1[key])
    return res_list


print(json.dumps(merge_json(json1, json2), indent=4))

How can I do that using terraform?

Answer 1

Terraform supports expanding a list into function parameters using the ... operator. This will allow an arbitrary number of documents to be read.

(I'm not sure, but I believe this feature was added in v0.15)

For this example, I added a new file 3.json with the contents:

[
  {
    "description": "description4_new",
    "url": "url4",
    "data": "data4_new"
  }
]

For main.tf, I'm using the same logic as @someguyonacomputer's answer:

$ cat main.tf

locals {
  jsondocs = [ 
    for filename in fileset(path.module, "*.json") : jsondecode(file(filename))
  ]

  as_dicts = [ 
    for arr in local.jsondocs : { 
      for obj in arr : obj.url => obj 
    }   
  ]

  # This is where the '...' operator is used
  merged = merge(local.as_dicts...)
}

output "as_list" {
  value = values(local.merged)
}

Result:

Changes to Outputs:
  + as_list = [
      + {
          + data        = "data1"
          + description = "description1"
          + url         = "url1"
        },
      + {
          + data        = "data2_new"
          + description = "description2_new"
          + url         = "url2"
        },
      + {
          + data        = "data3"
          + description = "description3"
          + url         = "url3"
        },
      + {
          + data        = "data4_new"
          + description = "description4_new"
          + url         = "url4"
        },
    ]

References:

• Terraform Docs -- Function Calls # Expanding Function Arguments

Answer 2

Using terraform 0.12.x

$ cat main.tf

locals {
  # read from files and turn into json
  list1 = jsondecode(file("1.json"))
  list2 = jsondecode(file("2.json"))
  # iterate over lists and turn url into a unique key
  dict1 = { for item in local.list1 : item.url => item }
  dict2 = { for item in local.list2 : item.url => item }
  # combine both dictionaries so values converge
  # only take its values
  merged = values(merge(local.dict1, local.dict2))
}

output "this" {
  value = local.merged
}

$ terraform apply

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

this = [
  {
    "data" = "data1"
    "description" = "description1"
    "url" = "url1"
  },
  {
    "data" = "data2_new"
    "description" = "description2_new"
    "url" = "url2"
  },
  {
    "data" = "data3"
    "description" = "description3"
    "url" = "url3"
  },
  {
    "data" = "data4"
    "description" = "description4"
    "url" = "url4"
  },
]