Create dictionary from list of strings using delimiter with Ansible

Question

I have a nested list of strings, which I am trying to convert into a dictionary. The list seems to be in a reasonable format, but my dictionary is getting overwritten each time I append to it.

Initial list:

TASK [test first_list] **********************************************************************************************************************************************************************************************************************************************************************************************************
ok: [node11] => {
    "first_list": [
        [
            "DEVNAME: /dev/sanstorage/node11-data103-2fd31b74e1ef5a8006c9ce9000b10399f",
            "UUID: 2751719f-d1a2-49b0-9e42-3d686c90d2e6",
            "TYPE: xfs"
        ],
        [
            "DEVNAME: /dev/sanstorage/node11-data104-2e81f1279171dce656c9ce9000b10399f",
            "UUID: 6a6265d6-b103-471e-9e25-e8cc5f5585a8",
            "TYPE: xfs"
        ],
        [
            "DEVNAME: /dev/sanstorage/node11-data102-2e385a327788d866e6c9ce9000b10399f",
            "UUID: 8c52d974-8584-4aa6-89b8-f1e1db016118",
            "TYPE: xfs"
        ],
        [
            "DEVNAME: /dev/sanstorage/node11-data101-241afe5ab93b5c0ee6c9ce9000b10399f",
            "UUID: 94b56164-6717-4b82-8f11-86fd94a39672",
            "TYPE: xfs"
        ],
        [
            "DEVNAME: /dev/sanstorage/node11-data100-25626d38d4c4239456c9ce9000b10399f",
            "UUID: 55a1a388-fe0a-4dd0-980a-a10c5317952e",
            "TYPE: xfs"
        ],
        [
            "DEVNAME: /dev/sanstorage/node11-data114-231d2661b7ab88f8f6c9ce9000b10399f",
            "UUID: f87ad708-1d12-41a5-9441-d32a97b5318c",
            "TYPE: ext4"
        ],
        [
            "DEVNAME: /dev/sanstorage/node11-data115-2d3a824975e90550f6c9ce9000b10399f",
            "UUID: b8b79886-9710-4205-900b-7b9d7d4ad933",
            "TYPE: ext4"
        ],
        [
            "DEVNAME: /dev/sanstorage/oneview-FA-data8165-284392eae1ad17d846c9ce9000b10399f",
            "UUID: c5a43057-676e-49b7-b2a9-b53a40f7010b",
            "TYPE: xfs"
        ],
        [
            "DEVNAME: /dev/sanstorage/oneview-FA-data3420-2e262703236f9b7046c9ce9000b10399f",
            "UUID: 1a70c187-9364-4f48-92f8-f9b9dec9824f",
            "TYPE: xfs"
        ],
        [
            "DEVNAME: /dev/sanstorage/node11-data112-2464954fd324508e66c9ce9000b10399f",
            "UUID: 2238a12e-2ca1-466e-8617-11051e1d612e",
            "TYPE: ext4"
        ],
        [
            "DEVNAME: /dev/sanstorage/node11-data111-25bb14827531149456c9ce9000b10399f",
            "UUID: db37479f-80b0-46b2-85e0-aef679bea164",
            "TYPE: ext4"
        ],
        [
            "DEVNAME: /dev/sanstorage/node11-data105-28ac6825aa80520d46c9ce9000b10399f",
            "UUID: e987fc7c-9a4f-46ea-91f0-4d5f37d3421e",
            "TYPE: xfs"
        ],
        [
            "DEVNAME: /dev/sanstorage/node11-data113-2177c50c4dd5063506c9ce9000b10399f",
            "UUID: 36c1194d-ac4f-4cee-8688-353974cb6be0",
            "TYPE: ext4"
        ]
    ]
}

From here, I try to form a dictionary, but only the last list entry is stored at the end:

TASK [test final_list] **********************************************************************************************************************************************************************************************************************************************************************************************************
ok: [node11] => {
    "final_list": {
        "DEVNAME": "/dev/sanstorage/node11-data113-2177c50c4dd5063506c9ce9000b10399f",
        "TYPE": "ext4",
        "UUID": "36c1194d-ac4f-4cee-8688-353974cb6be0"
    }
}

Ideally, it would look something like this, I'm guessing I need to nest the dictionary within a list?

    "final list": [
        {
            "DEVNAME: /dev/sanstorage/node11-data103-2fd31b74e1ef5a8006c9ce9000b10399f",
            "UUID: 2751719f-d1a2-49b0-9e42-3d686c90d2e6",
            "TYPE: xfs"
        },
        {
            "DEVNAME: /dev/sanstorage/node11-data104-2e81f1279171dce656c9ce9000b10399f",
            "UUID: 6a6265d6-b103-471e-9e25-e8cc5f5585a8",
            "TYPE: xfs"
        },
        {
            "DEVNAME: /dev/sanstorage/node11-data102-2e385a327788d866e6c9ce9000b10399f",
            "UUID: 8c52d974-8584-4aa6-89b8-f1e1db016118",
            "TYPE: xfs"
        },
... snip ...

Below is the important part of the playbook:

    - set_fact:
        first_list: "{{ first_list | default([]) + [disk_data.split('\n')] }}"
      vars:
        disk_data: '{{ item.stdout }}'        
      with_items: "{{ disk_check.results }}"

    - name: test first_list
      debug:
        var: first_list
    
    - set_fact: final_list:{}

    - set_fact:
        final_list: "{{ final_list | default([]) | combine(dict([ item.partition(':')[::2]|map('trim')])) }}"
      with_items: "{{ first_list }}"

    - name: test final_list
      debug:
        var: final_list

Thoughts?

Answer 1

TL;DR

- debug:
    msg: "{{ disk_check.results | map(attribute='stdout') | map('from_yaml') | list }}"

---

You did not provide the initial task which returns disk_check. From your usage in your example, I'll take for granted it is a shell or command task used in a loop.

Meanwhile you still provided enough information so we can avoid falling into an x/y problem. Your question is basically "how can I split a string on delimiters and use the result to create a dict?" where what you really want is more "How can I parse a string representing a yaml dict into an actual dict?"

To start with

- set_fact:
    first_list: "{{ first_list | default([]) + [disk_data.split('\n')] }}"
  vars:
    disk_data: '{{ item.stdout }}'        
  with_items: "{{ disk_check.results }}"

This is going through all your command results, spliting stdout on a new line and adding that list to a top level list.

Since the output of shell/command contains a stdout_lines attribute and that we can extract attributes from a list of dicts with the jinja2 map filter, this could be replaced in one go without having to run a task at all with the following jinja expression:

"{{ disk_check.results | map(attribute='stdout_lines') | list }}"

But we would still be walking the wrong path.

Let's have a look at one of your (reconstructed) individual stdout from results

"stdout": "DEVNAME: /dev/sanstorage/node11-data102-2e385a327788d866e6c9ce9000b10399f\nUUID: 8c52d974-8584-4aa6-89b8-f1e1db016118\nTYPE: xfs"

This is a string representation of a yaml dict. Ansible has a from_yaml filter. And we can use the map filter to apply a filter to each element in a list.

The below playbook tries to reproduce your original data to display it in one go

---
- name: Parse a yaml dict in each element of a list
  hosts: localhost
  gather_facts: false

  vars:
    example_disks: 3

  tasks:

    - name: Faking your disk check supposed command
      shell: |-
        cat <<EOF
        DEVNAME: /dev/sanstorage/node{{ (item | string)*2 }}-data{{ (item | string)*3 }}-$(uuidgen)
        UUID: $(uuidgen)
        TYPE: xfs
        EOF
      loop: "{{ range(1, example_disks+1) }}"
      register: disk_check

    - name: Show the original data (use -v to trigger)
      debug:
        var: disk_check
        verbosity: 1

    - name: Display a list of dicts from the above result
      debug:
        msg: "{{ disk_check.results | map(attribute='stdout') | map('from_yaml') | list }}"

and gives (run with -v to show the intermediate debug):

PLAY [Parse a yaml dict in each element of a list] *************************************************************************************************************************************************************************************

TASK [Faking your disk check supposed command] *****************************************************************************************************************************************************************************************
changed: [localhost] => (item=1)
changed: [localhost] => (item=2)
changed: [localhost] => (item=3)

TASK [Show the original data (use -v to trigger)] **********************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [Display a list of dicts from the above result] ***********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": [
        {
            "DEVNAME": "/dev/sanstorage/node11-data111-10e7a318-62af-46db-895e-5d94b0c2cf88",
            "TYPE": "xfs",
            "UUID": "5a87ae1c-c312-4325-88ac-b9cc1edfa69b"
        },
        {
            "DEVNAME": "/dev/sanstorage/node22-data222-18247d1d-87b2-4c7d-8d48-cd333d7530f9",
            "TYPE": "xfs",
            "UUID": "bc0d7f1a-16e4-4694-b3e2-904e69cc208d"
        },
        {
            "DEVNAME": "/dev/sanstorage/node33-data333-21bc7fde-645f-4cf2-9e18-72d004700085",
            "TYPE": "xfs",
            "UUID": "58985028-3eb1-4f34-90e9-f0e4c8257607"
        }
    ]
}

PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0 

Answer 2

Let's simplify the data, e.g.

  first_list:
  - - a: 1
    - b: 1
  - - a: 2
    - b: 2

The expected result is

  final_list:
  - a: 1
    b: 1
  - a: 2
    b: 2

The task below does the job

    - set_fact:
        final_list: "{{ final_list|default([]) + [_dict|from_yaml] }}"
      loop: "{{ first_list }}"
      vars:
        _dict: |
          {% for i in item %}
          {{ (i|to_yaml)[1:-2] }}
          {% endfor %}

There are other options on how to format the item, e.g.

          {{ i.keys()|first }}: {{ i.values()|first }}

---

Write a filter if you use such conversions frequently, e.g.

shell> cat filter_plugins/list_filters.py
def list2dict(l):
    out = []
    for i in l:
        item = {}
        for j in range(0, len(i)):
            item.update(i[j])
        out.append(item)
    return out


class FilterModule(object):
    ''' List filters. '''

    def filters(self):
        return {
            'list2dict': list2dict,
        }

Then the simple filter below gives the same result

    - set_fact:
        final_list: "{{ first_list|list2dict }}"