ansible merge two lists based on an attribute

Question

I have a role in my playbook that generates two lists using set_fact. The two facts are used in different tasks. I then need to merge them for a final task.

list1:
  - name: alice
    roles: ['role1', 'role2']
  - name: bob
    roles: ['role1']

list2:
  - name: alice
    roles: ['role3']
  - name: charlie
    roles: ['role2']

For my final task I need the output to be:

list3:
  - name: alice
    roles: ['role1', 'role2', 'role3']
  - name: bob
    roles: ['role1']
  - name: charlie
    roles: ['role2']

Answer 1

This is a solution in plain ansible. If it becomes out of control because of your datastructure growth, you should consider writing your own filter (example)

---
- name: demo playbook for deeps dictionary merge
  hosts: localhost
  gather_facts: false

  vars:
    list1:
      - name: alice
        roles: ['role1', 'role2']
      - name: bob
        roles: ['role1']

    list2:
      - name: alice
        roles: ['role3']
      - name: charlie
        roles: ['role2']

  tasks:

    - debug:
        msg: >-
          roles for {{ item }}:
          {{
            (list1 | json_query("[?name == '" + item +"'].roles")).0 | default([])
            +
            (list2 | json_query("[?name == '" + item +"'].roles")).0 | default([])
          }}
      loop: >-
        {{
          (
            list1 | json_query("[].name")
            +
            list2 | json_query("[].name")
          )
          | unique
        }}

Which gives:

PLAY [demo playbook for deeps dictionary merge] ************************************************************************************************************************************************************************************************************************

TASK [debug] ************************************************************************************************************************************************************************************************************************************************************
Wednesday 24 April 2019  16:23:07 +0200 (0:00:00.046)       0:00:00.046 ******* 
ok: [localhost] => (item=alice) => {
    "msg": "roles for alice: ['role1', 'role2', 'role3']"
}
ok: [localhost] => (item=bob) => {
    "msg": "roles for bob: ['role1']"
}
ok: [localhost] => (item=charlie) => {
    "msg": "roles for charlie: ['role2']"
}

Answer 2

I asked about lists vs dictionaries in the comment because of the impact it will have on the solution. If you were to restructure your data like this:

dict1:
  alice: ['role1', 'role2']
  bob: ['role1']

dict2:
  alice: ['role3']
  charlie: ['role2']

Then your solution becomes:

- set_fact:
    dict3: >-
      {{
      dict3|default([])|combine({
      item: (dict1[item]|default([]) + dict2[item]|default([]))|unique
      })
      }}
  loop: "{{ (dict1.keys()|list + dict2.keys()|list)|unique }}"

- debug:
    var: dict3

Which outputs:

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "dict3": {
        "alice": [
            "role1", 
            "role2", 
            "role3"
        ], 
        "bob": [
            "role1"
        ], 
        "charlie": [
            "role2"
        ]
    }
}

---

If you're stuck with using lists, we can improve upon the json_query solution that Zeitounator suggested:

- set_fact:
    list3: >-
      {{
      list3|default([]) + [{
      'name': item,
      'roles': (list1|json_query('[?name==`' + item + '`].roles[]') + list2|json_query('[?name==`' + item + '`].roles[]'))|unique
      }]
      }}
  loop: "{{ (list1|json_query('[].name') + list2|json_query('[].name'))|unique }}"

- debug:
    var: list3

This produces your desired output:

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "list3": [
        {
            "name": "alice", 
            "roles": [
                "role1", 
                "role2", 
                "role3"
            ]
        }, 
        {
            "name": "bob", 
            "roles": [
                "role1"
            ]
        }, 
        {
            "name": "charlie", 
            "roles": [
                "role2"
            ]
        }
    ]
}