How to include variables with include_vars with the same name without overwriting previous

Question

I am having this let's call it include.yaml

#- name: "Playing with Ansible and Include files"
- hosts: localhost
  connection: local 
  tasks:
    - find: paths="./" recurse=yes patterns="test.yaml"
      register: file_to_exclude
    - debug: var=file_to_exclude.stdout_lines        
    - name: shell
      shell: "find \"$(pwd)\" -name 'test.yaml'"
      register: files_from_dirs     
    - debug: var=files_from_dirs.stdout_lines
    - name: Include variable files
      include_vars: "{{ item }}"
      with_items:
        - "{{ files_from_dirs.stdout_lines }}"
    - debug: var=files 

and 2 ore more test files

./dir1/test.yaml

that contains

files: 
  - file1 
  - file2

./dir2/test.yaml

that contains

files: 
  - file3 
  - file4

the result is

TASK [Include variable files] ******************************************************************************************
ok: [localhost] => (item=/mnt/c/Users/GFlorinescu/ansible_scripts/ansible/1st/test.yaml)
ok: [localhost] => (item=/mnt/c/Users/GFlorinescu/ansible_scripts/ansible/2nd/test.yaml)

TASK [debug] ***********************************************************************************************************
ok: [localhost] => {
    "files": [
        "file3",
        "file4"
    ]
}

How can I get all the values in files, at the moment the last included files variable from last file overrides the files from the previous files? Of course without changing the variables names in files test.yaml?

In other words I want files to be:

ok: [localhost] => {
    "files": [
        "file1",
        "file2",
        "file3",
        "file4"
    ]
}

To be more specific, I ask for any kind of solution or module, even not official or some github module, I don't want a specific include_vars module solution.

Answer 1

Put the included variables into the dictionaries with unique names. For example, create the names from the index of the loop. Then, iterate the names and concatenate the lists

    - command: "find {{ playbook_dir }} -name test.yaml"
      register: files_from_dirs

    - include_vars:
        file: "{{ item }}"
        name: "{{ name }}"
      loop: "{{ files_from_dirs.stdout_lines }}"
      loop_control:
        extended: true
      vars:
        name: "files_{{ ansible_loop.index }}"

    - set_fact:
        files: "{{ files|d([]) + lookup('vars', item).files }}"
      with_varnames: "files_[0-9]+"

    - debug:
        var: files

give

files:
  - file1
  - file2
  - file3
  - file4

---

Notes:

• You have to provide either a path relative to the home directory or an absolute path. See the example below

    - command: "echo $PWD"
      register: out
    - debug:
        var: out.stdout

give

out.stdout: /home/admin

For example, when you want to find the files relative to the directory of the playbook

    - command: "find {{ playbook_dir }} -name test.yaml"
      register: files_from_dirs     
    - debug:
        var: files_from_dirs.stdout_lines

give

files_from_dirs.stdout_lines:
  - /export/scratch/tmp8/test-987/dir1/test.yaml
  - /export/scratch/tmp8/test-987/dir2/test.yaml

• The same is valid for the module find. For example,

    - find:
        paths: "{{ playbook_dir }}"
        recurse: true
        patterns: test.yaml
      register: files_from_dirs
    - debug:
        var: files_from_dirs.files|map(attribute='path')|list

give the same result

files_from_dirs.files|map(attribute='path')|list:
  - /export/scratch/tmp8/test-987/dir1/test.yaml
  - /export/scratch/tmp8/test-987/dir2/test.yaml

• Simplify the code and put the declaration of files into the vars. For example, the below declaration gives the same result

files: "{{ query('varnames', 'files_[0-9]+')|
           map('extract', hostvars.localhost, 'files')|
           flatten }}"

Example of a complete playbook for testing

- hosts: localhost

  vars:

    files: "{{ query('varnames', 'files_[0-9]+')|
               map('extract', hostvars.localhost, 'files')|
               flatten }}"

  tasks:

    - find:
        paths: "{{ playbook_dir }}"
        recurse: true
        patterns: test.yaml
      register: files_from_dirs

    - include_vars:
        file: "{{ item }}"
        name: "{{ name }}"
      loop: "{{ files_from_dirs.files|map(attribute='path')|list }}"
      loop_control:
        extended: true
      vars:
        name: "files_{{ ansible_loop.index }}"

    - debug:
        var: files

---

(maybe off-topic, see comments)

Q: "Is there a way to write the path where it was found?"

A: Yes, it is. See the self-explaining example below. Given the inventory

shell> cat hosts
host_1 file_1=alice
host_2 file_2=bob
host_3

the playbook

- hosts: host_1,host_2,host_3

  vars:

    file_1_list: "{{ hostvars|json_query('*.file_1') }}"
    file_2_list: "{{ hostvars|json_query('*.file_2') }}"
    file_1_dict: "{{ dict(hostvars|dict2items|
                          selectattr('value.file_1', 'defined')|
                          json_query('[].[key, value.file_1]')) }}"
    file_1_lis2: "{{ hostvars|dict2items|
                     selectattr('value.file_1', 'defined')|
                     json_query('[].{key: key, file_1: value.file_1}') }}"

  tasks:

    - debug:
        msg: |-
          file_1_list: {{ file_1_list }}
          file_2_list: {{ file_2_list }}
          file_1_dict:
            {{ file_1_dict|to_nice_yaml|indent(2) }}
          file_1_lis2:
            {{ file_1_lis2|to_nice_yaml|indent(2) }}
      run_once: true

gives

  msg: |-
    file_1_list: ['alice']
    file_2_list: ['bob']
    file_1_dict:
      host_1: alice
  
    file_1_lis2:
      -   file_1: alice
          key: host_1