Looping over a nested ansible dict

Question

I have a given vars as a list of kafka topics and possible configurations, like:

kafka_topics:
  foo:
    retentiontime: 3600
    deletepolicy: delete
  bar:
    retentiontime: 3600
    compression: gzip

I have figured multiple ways with dict2items to set them explicit like:

    - name: Set RetentionTime for Topics 
      debug:
        msg: "Topic {{ item.key }} get {{ item.value.retentiontime }} for retentiontime "
      loop: "{{ lookup('dict', kafka_topics) }}"
      when: item.value.retentiontime is defined

But is it possible to get a output like

Topic foo get 3600 for retentiontime
Topic foo get delete for deletepolicy
Topic bar get 3600 for retentiontime
Topic bar get gzip for compression

without defining the value name by name with ansible?

I also tried with

    - name: Loop over subelements of the dictionary
      debug:
        msg: "Key={{ item.0.key }} value={{ item.1 }}"
      loop: "{{ lookup('dict', kafka_topics) | list | subelements('value') }}"

which prints Key=bar value={'compression': gzip} but now i'm stuck at seperating those two. Is there a way to extract from item.1 the key and the value?

Answer 1

@Vladimir's answer is indeed the most straightforward one if you need to write some debugging info as above or create a configuration file from a template.

But this might get a little trickier if you actually need to loop over this data in a classic task where you have to pass each value separately to the corresponding moldule's option.

In such cases, here is an alternative that will transform your current dict to list.

The goal is to go from:

"first topic name":
  "option 1 name": "option 1 value"
  "option 2 name": "option 2 value"
"second topic name": ...

to

- topic_name: "first topic name"
  topic_options:
    - option_name: "option 1 name"
      option_value: "option 1 value"
    - option_name: "option 2 name"
      option_value: "option 2 value"
- topic_name: "second topic name"
  ...

This transformed data is then easily loopable with subelements as demonstrated in the following playbook:

---
- hosts: localhost
  gather_facts: false

  vars:
    kafka_topics: {"foo": {"retentiontime": 3600, "deletepolicy": "delete"}, "bar": {"retentiontime": 3600, "compression": "gzip"}}

  tasks:
    - name: Transform our list to something easier to loop with subelements
      vars:
        current_topic:
          topic_name: "{{ item.topic_name }}"
          topic_options: "{{ item.topic_options | dict2items(key_name='option_name', value_name='option_value') }}"
      set_fact:
        my_topic_list: "{{ my_topic_list | default([]) + [current_topic] }}"
      loop: "{{ kafka_topics | dict2items(key_name='topic_name', value_name='topic_options') }}"

    - name: Show the transformed var
      debug:
        var: my_topic_list

    - name: Loop over topics and their options
      debug:
        msg: "Topic `{{ topic.0.topic_name }}` has option `{{ topic.1.option_name }}` with value `{{ topic.1.option_value }}`"
      loop: "{{ my_topic_list | subelements('topic_options') }}"
      loop_control:
        label: "{{ topic.0.topic_name }} - {{ topic.1.option_name }} - {{ topic.1.option_value }}"
        loop_var: topic

Which gives:

PLAY [localhost] ***********************************************************************************************************************************************************************************************************************

TASK [Transform our list to something easier to loop with subelements] *****************************************************************************************************************************************************************
ok: [localhost] => (item={'topic_name': 'foo', 'topic_options': {'retentiontime': 3600, 'deletepolicy': 'delete'}})
ok: [localhost] => (item={'topic_name': 'bar', 'topic_options': {'retentiontime': 3600, 'compression': 'gzip'}})

TASK [Show the transformed var] ********************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "my_topic_list": [
        {
            "topic_name": "foo",
            "topic_options": [
                {
                    "option_name": "retentiontime",
                    "option_value": 3600
                },
                {
                    "option_name": "deletepolicy",
                    "option_value": "delete"
                }
            ]
        },
        {
            "topic_name": "bar",
            "topic_options": [
                {
                    "option_name": "retentiontime",
                    "option_value": 3600
                },
                {
                    "option_name": "compression",
                    "option_value": "gzip"
                }
            ]
        }
    ]
}

TASK [Loop over topics and their options] **********************************************************************************************************************************************************************************************
ok: [localhost] => (item=foo - retentiontime - 3600) => {
    "msg": "Topic `foo` has option `retentiontime` with value `3600`"
}
ok: [localhost] => (item=foo - deletepolicy - delete) => {
    "msg": "Topic `foo` has option `deletepolicy` with value `delete`"
}
ok: [localhost] => (item=bar - retentiontime - 3600) => {
    "msg": "Topic `bar` has option `retentiontime` with value `3600`"
}
ok: [localhost] => (item=bar - compression - gzip) => {
    "msg": "Topic `bar` has option `compression` with value `gzip`"
}

PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Answer 2

The simplest option is Jinja. For example

    - debug:
        msg: |-
          {% for k1,v1 in kafka_topics.items() %}
          {% for k2,v2 in v1.items() %}
          Topic {{ k1 }} get {{ v2 }} for {{ k2 }}
          {% endfor %}
          {% endfor %}

gives

  msg: |-
    Topic foo get 3600 for retentiontime
    Topic foo get delete for deletepolicy
    Topic bar get 3600 for retentiontime
    Topic bar get gzip for compression

---

Create a dictionary of lists first if the iteration is needed. For example the template

shell> cat templates/kafka_facts_list.j2 
{% for k1,v1 in kafka_topics.items() %}
{{ k1 }}:
{% for k2,v2 in v1.items() %}
  - {key: {{ k2 }}, val: {{ v2 }}}
{% endfor %}
{% endfor %}

    - set_fact:
        dict2: "{{ lookup('template', 'kafka_facts_list.j2')|from_yaml }}"
    - debug:
        var: dict2

gives

  dict2:
    bar:
    - key: retentiontime
      val: 3600
    - key: compression
      val: gzip
    foo:
    - key: retentiontime
      val: 3600
    - key: deletepolicy
      val: delete

Then use subelements

    - debug:
        msg: "Topic {{ item.0.key }} get {{ item.1.val }} for {{ item.1.key }}"
      with_subelements:
        - "{{ dict2|dict2items }}"
        - value

gives

  msg: Topic foo get 3600 for retentiontime
  msg: Topic foo get delete for deletepolicy
  msg: Topic bar get 3600 for retentiontime
  msg: Topic bar get gzip for compression