Ansible dict: How to group by a key and set fact to be re-used later

Question

I've a CSV which I converted into a dict using read_csv module. My aim is to group the data by a field within the dictionary. For example, in below data I want to use firstname & secondname group by departmentGroup, so that I get a dict array to be re-used later

Below is the input dict created from csv

      {
        {
            "ID02": {
                "department": "IT",
                "departmentGroup": "IT-development",
                "firstname": "first02",
                "id": "ID02",
                "salary": "40000",
                "secondname": "surnam2",
                "subDepartment": "development"
            }
        },
        {
            "ID03": {
                "department": "IT",
                "departmentGroup": "IT-development",
                "firstname": "first03",
                "id": "ID03",
                "salary": "42000",
                "secondname": "surnam3",
                "subDepartment": "development"
            }
        },
        {
            "ID04": {
                "department": "IT",
                "departmentGroup": "IT-operations",
                "firstname": "first04",
                "id": "ID04",
                "salary": "46000",
                "secondname": "surnam4",
                "subDepartment": "operations"
            }
        },
        {
            "ID05": {
                "department": "IT",
                "departmentGroup": "IT-operations",
                "firstname": "first05",
                "id": "ID05",
                "salary": "42000",
                "secondname": "surnam5",
                "subDepartment": "operations"
            }
        }

I'm looking for output of values grouped by departmentGroup, so i can loop it for future tasks to build-up templates

{ 
  "IT-development": 
    [
      {"id": "ID02", "firstname": "first02", "secondname": "surnam2"},
      {"id": "ID03", "firstname": "first03", "secondname": "surnam3"}    
    ],
  "IT-operations": 
    [
      {"id": "ID04", "firstname": "first04", "secondname": "surnam4"},
      {"id": "ID05", "firstname": "first05", "secondname": "surnam5"}
    ]
}

I've tried using map but couldn't reach much. Any ideas on how to group in such a way in ansible?

Answer 1

I confirmed that I could do it by using the filters only.
The below is simpler than using Jinja2 code.

---
- name: Example Playbook for 65447303
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Read a csv file and converts to dict
      read_csv:
        path: test.csv
        key: name
      register: csv_result

    - debug: msg="{{ csv_result }}"

    - name: Set combined data variable
      set_fact:
        combined_data: >-
          {{ combined_data | default({})
            | combine({item.value.departmentGroup: []
            + [{
                'id': item.value.id,
                'firstname': item.value.firstname,
                'secondname': item.value.secondname
              }]
            + combined_data[item.value.departmentGroup] | default([]) })
          }}
      with_dict: "{{ csv_result.dict }}"

    - debug: msg="{{ combined_data }}"

Answer 2

I could do it by using Jinja2.
It wasn't easy to do it using the Ansible or Jinja2 filters only.
So I implemented the code using Jinja2.
How about this like?

csv file

(venv) $ vi test.csv

name,department,departmentGroup,firstname,id,salary,secondname,subDepartment
ID02,IT,IT-development,first02,ID02,40000,surnam2,development
ID03,IT,IT-development,first03,ID03,42000,surnam3,development
ID04,IT,IT-operations,first04,ID04,46000,surnam4,operations
ID05,IT,IT-operations,first05,ID05,42000,surnam5,operations

Playbook

(venv) $ vi main.yml

---
- name: Example Playbook for 65447303
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Read a csv file and converts to dict
      read_csv:
        path: test.csv
        key: name
      register: csv_result

    - debug: msg="{{ csv_result }}"

    - name: Set combined data variable
      set_fact:
        combined_data: |
          {% set data = {} %}
          {% set tmp = {} %}
          {# initialize data variable #}
          {% for key in csv_result.dict.keys() %}
          {%     set _ = tmp.update(csv_result.dict[key]) %}
          {%     set _ = data.update({tmp['departmentGroup']: []}) %}
          {# Append the data to each department key #}
          {% endfor %}
          {% for key in csv_result.dict.keys() %}
          {%     set _ = tmp.update(csv_result.dict[key]) %}
          {%     set _ = data[tmp['departmentGroup']].append({
                           'id': tmp['id'],
                           'firstname': tmp['firstname'],
                           'secondname': tmp['secondname']
                         }) %}
          {% endfor %}
          {{ data }}

    - debug: msg="{{ combined_data | type_debug }}"

    - debug: msg="{{ combined_data }}"

Execute and Result

(venv) $ ansible-playbook main.yml

(snip)
TASK [debug] ***************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": {
        "IT-development": [
            {
                "firstname": "first02",
                "id": "ID02",
                "secondname": "surnam2"
            },
            {
                "firstname": "first03",
                "id": "ID03",
                "secondname": "surnam3"
            }
        ],
        "IT-operations": [
            {
                "firstname": "first04",
                "id": "ID04",
                "secondname": "surnam4"
            },
            {
                "firstname": "first05",
                "id": "ID05",
                "secondname": "surnam5"
            }
        ]
    }
}
(snip)