Ansible merge different size list which has key value pairs

Question

I have two list:

list1:
  - file: f1
    perm: '777'
  - file: f2
    perm: '677'
  - file: f3
    perm: '755'
  - file: f4
    perm: '700'

list2:
  - file: f4
    t_perm: '755'
  - file: f3
    perm: '677'

I have to merge them into new list as below:

list3:
  - file: f1
    perm: '777'
  - file: f2
    perm: '677'
  - file: f3
    perm: '755'
    t_perm: '677'
  - file: f4
    perm: '700'
    t_perm: '755'

I tried as below, but it did not merge correctly.

  - name: merge the list 
    set_fact:
     list3: "{{ list3 + [ item[0] | combine(item[1]) ]  }}"
    when: item[0].file == item[1].file
    loop: "{{ list1(sort(attribute='file')| zip(list2|sort(attribute='file') | list}}"

Additinally, if possible can list3 can have an extra field that the target file is missing?

list3:
  - file: f1
    perm: '777'
    stat: not_found
  - file: f2
    perm: '677'
    stat: not_found
  - file: f3
    perm: '755'
    t_perm: '677'
  - file: f4
    perm: '700'
    t_perm: '755'

Answer 1

Update.

The filter community.general.lists_mergeby does the job. Given the lists

  list1:
    - {file: f1, perm: '777'}
    - {file: f2, perm: '677'}
    - {file: f3, perm: '755'}
    - {file: f4, perm: '700'}

  list2:
    - {file: f4, t_perm: '755'}
    - {file: f3, t_perm: '677'}

Declare list3

  list3: "{{ [list1, list2]|community.general.lists_mergeby('file') }}"

gives the expected result

  list3:
    - {file: f1, perm: '777'}
    - {file: f2, perm: '677'}
    - {file: f3, perm: '755', t_perm: '677'}
    - {file: f4, perm: '700', t_perm: '755'}

---

Example of a complete playbook for testing

- hosts: localhost

  vars:

    list1:
      - {file: f1, perm: '777'}
      - {file: f2, perm: '677'}
      - {file: f3, perm: '755'}
      - {file: f4, perm: '700'}
    list2:
      - {file: f4, t_perm: '755'}
      - {file: f3, t_perm: '677'}

    list3: "{{ [list1, list2]|community.general.lists_mergeby('file') }}"

  tasks:

    - debug:
        var: list1|to_yaml
    - debug:
        var: list2|to_yaml
    - debug:
        var: list3|to_yaml

---

Original.

The tasks

    - set_fact:
        list3: "{{ (list1 + list2)|
                   groupby('file')|
                   map('last')|
                   map('combine')|
                   list }}"
    - debug:
        var: list3

give

  list3:
  - file: f1
    perm: '777'
  - file: f2
    perm: '677'
  - file: f3
    perm: '755'
    t_perm: '677'
  - file: f4
    perm: '700'
    t_perm: '755'

Answer 2

I think the easiest path forward is first converting list1 into a dictionary, then merging in the elements of list2, like this:

- hosts: localhost
  gather_facts: false
  vars:
    list1:
      - file: f1
        perm: '777'
      - file: f2
        perm: '677'
      - file: f3
        perm: '755'
      - file: f4
        perm: '700'
    list2:
      - file: f4
        t_perm: '755'
      - file: f3
        perm: '677'

  tasks:
    - name: convert list1 into a dictionary
      set_fact:
        d1: "{{ d1|combine({item.file: item}) }}"
      loop: "{{ list1 }}"
      vars:
        d1: {}

    - name: update d1 with the values from list2
      set_fact:
        d1: "{{ d1|combine({item.file: item}, recursive=true) }}"
      loop: "{{ list2 }}"
    
    - name: set list3 to the values of d1
      set_fact:
        list3: "{{ d1.values()|list }}"

    - debug:
        var: list3

The output of the above is:

TASK [debug] *********************************************************************************************************************************************************************************
ok: [localhost] => {
    "list3": [
        {
            "file": "f1",
            "perm": "777"
        },
        {
            "file": "f2",
            "perm": "677"
        },
        {
            "file": "f3",
            "perm": "677"
        },
        {
            "file": "f4",
            "perm": "700",
            "t_perm": "755"
        }
    ]
}

...which is I think what you were looking for.