How to "slurp" multiple files with Ansible

Question

With Ansible, I need to extract content from multiple files. With one file I used slurp and registered a variable.

- name: extract nameserver from .conf file
  slurp:
    src: /opt/test/roles/bind_testing/templates/zones/example_file.it.j2
  delegate_to: 127.0.0.1
  register: file

- debug:
   msg: "{{ file['content'] | b64decode }}"

But now I have multiple files so I need to extract content from every files, register them one by one to be able to process them later with operations like sed, merging_list etc...

How can I do this in ansible?

I tried to use slurp with with_fileglob directive but I couldn't register the files...

- name: extract nameserver from .conf file
  slurp:
    src: "{{ item }}"
  with_fileglob:
    - "/opt/test/roles/bind9_domain/templates/zones/*"
  delegate_to: 127.0.0.1
  register: file

Answer 1

Performance issues with slurping lots of files

The approch developed in @guzmonne's answer is perfectly correct, but can be VERY slow if you are planning to slurp lots of files. Essentialy because using a loop forces Ansible to replay a large amount of python code, including internal actions in addition to the slurp module code.

Solution : a custom module

In such situations, the most efficient approach consists in forking the module you playbook relies on, to adapt it's behaviour on what you want to do, which here is slurping an entire directory.

There are several places where you can put the forked module. In the following example, I chose to put it in a local collection because RedHat's Ansible VS Code plugin uses them for IDE feature (auto-completion, helpers, etc...) I named the collection local.tools, and the module sluuurp.

$ mkdir -p collections/ansible_collections
$ ansible-galaxy collection init --init-path collections/ansible_collections local.tools

Now put the following code (way inspired from ansible.builtin.slurp module) into collections/ansible_collections/local/tools/plugins/modules/sluuurp.py

from __future__ import absolute_import, division, print_function
__metaclass__ = type

DOCUMENTATION = r'''
---
module: sluuurp
short_description: Slurps files from a directory on a remote node
description:
     - This module aims to provide a simple way to slurps multiple files given a folder path where they are located.
     - An optional regex can be used to filter file names under the targeted directory.
options:
  dir:
    description:
      - The path of the directory on the remote system to fetch.
    type: path
    required: true
  match:
    description:
      - The regex filter for selecting files.
    type: str
    required: false
attributes:
  check_mode:
    support: full
  diff_mode:
    support: none
  platform:
    platforms: posix
notes:
   - This module will returns a key-value dict where keys matches slurped file names and values matches files content, base64 encoded.
seealso:
- module: ansible.builtin.slurp
author:
    - fizzgig1888
'''

EXAMPLES = r'''
- name: Slurps all files in /etc/nginx
  ansible.builtin.sluuurp:
    dir: /etc/nginx
  register: slurped

- name: Slurps conf files in /etc/nginx
  ansible.builtin.sluuurp:
    dir: /etc/nginx
    match: ^.+\.conf$
  register: slurped
'''

RETURN = r'''
contents:
  description: Filename-Filecontent (base64 encoded) key-value dictionary
  returned: success
  type: dict
dir:
  description: The dir path parameter, returned as a copy
  returned: success
  type: path
'''

from base64 import b64encode
import errno
import re
from os import listdir
from os.path import isfile, isdir, join, basename

from ansible.module_utils.basic import AnsibleModule

def main():
  module = AnsibleModule(
    argument_spec=dict(
      dir=dict(type='path', required=True),
      match=dict(type='str', required=False)
    ),
    supports_check_mode=True,
  )
  dir = module.params['dir']

  if not isdir(dir):
    module.fail_json(f"provided \"dir\" is not a directory : {dir}")

  if module.params['match'] is not None:
    match = module.params['match']
    try:
      reg = re.compile(match)
    except re.error:
      module.fail_json(f"provided \"match\" field is not a valid regex : {match}")
    
  # List matching files in directory
  if module.params['match'] is not None:
    paths = [join(dir, f) for f in listdir(dir) if isfile(join(dir, f)) and reg.search(f)]
  else:
    paths = [join(dir, f) for f in listdir(dir) if isfile(join(dir, f))]

  contents = {}
  for p in paths:
    try:
      with open(p, 'rb') as src:
        content = src.read()
        src.close()
    except OSError as e:
      if e.errno == errno.EACCES:
        msg = f"failed to read file : {p}. Check permissions."
      else:
        msg = f"other error when trying to slurp : {p}"
      module.fail_json(msg)

    contents[basename(p)]=b64encode(content)

  module.exit_json(contents=contents, dir=dir)

if __name__ == '__main__':
  main()

What does this module do ?

It's described in the global vars of the module ifself. But in a few words, this sluuurp.py module slurps the files in a managed node directory and builds a dictionary which keys are file names in this directory, and values are the file contents base64 encoded.

It also implements a simple mecanism to filter which files are meant to be slurped, with a regex check on their names.

Test it yourself !

Create sample files on your ansible managed node, in the home directory on the ansible_ssh_user (or the ansible_sudo_user, if you plan to run the sluuurp task using become: true).

$ mkdir ~/bigdir
$ for i in {01..99}; do { echo content$i | tee ~/bigdir/file$i; }; done

Now sluuurps them, and debug the content using the two following tasks in an ansible play :

- name: Sluuurp files
  local.tools.sluuurp:
    dir: "~/bigdir"
  register: slurped

- name: Debug slurped
  ansible.builtin.debug:
    msg: "{{ slurped['contents'] }}"

Which outputs this result :

TASK [Sluuurp files] ******************************************************************************************************************************
ok: [node]

TASK [Debug slurped] ******************************************************************************************************************************
ok: [node] => {
    "msg": {
        "file01": "Y29udGVudDAxCg==",
        "file02": "Y29udGVudDAyCg==",
        "file03": "Y29udGVudDAzCg==",
        "file04": "Y29udGVudDA0Cg==",
        ...
        "file97": "Y29udGVudDk3Cg==",
        "file98": "Y29udGVudDk4Cg==",
        "file99": "Y29udGVudDk5Cg=="
    }
}
PLAY RECAP *************************************************************************************************************************************************
node                 : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Additional example

Let's sluuurp only files from 90 to 99 using a regex. And let's display their content in plain text (base64 decoded).

The two ansible tasks :

- name: Sluuurp files
  local.tools.sluuurp:
    dir: "~/bigdir"
    match: "9[0-9]"
  register: slurped

- name: Debug slurped
  ansible.builtin.debug:
    msg: "{{ dict(contents.keys() | zip(contents.values() | map('b64decode'))) }}"
  vars:
    contents: "{{ slurped['contents'] }}"

The output :

TASK [Sluuurp files] ******************************************************************************************************************************
ok: [node]

TASK [Debug slurped] ******************************************************************************************************************************
ok: [node] => {
    "msg": {
        "file90": "content90\n",
        "file91": "content91\n",
        "file92": "content92\n",
        "file93": "content93\n",
        "file94": "content94\n",
        "file95": "content95\n",
        "file96": "content96\n",
        "file97": "content97\n",
        "file98": "content98\n",
        "file99": "content99\n"
    }
}
PLAY RECAP *************************************************************************************************************************************************
node                 : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Answer 2

If you're looking for a host's mounts you could utilize ansible_facts.

    "ansible_mounts": example:

- name: Free-space.yml Date
  command: date

- name: Display more vars
  debug:
    msg:
    - "Variable list"
    - "diskfree_req is:  '{{diskfree_req}}' "
    - "mountname is:     '{{mountname}}' "

- name: PreTest for freespace
  vars: 
    deprecation_warnings: False
    #mountname: '/opt/oracle'
    mount: "{{ ansible_mounts | selectattr('mount','equalto', mountname) | first }}"

  assert:
    that: mount.size_available > {{diskfree_req}}
    msg: 
    - "DANGER : disk space is low"
    - "'{{mountname}} only has {{mount.size_available}} available. Please correct"
  register: disk_free

Answer 3

You should just use the loop option, configured with the list of files to slurp. Check this example:

---
- hosts: local
  connection: local
  gather_facts: no
  tasks:
    - name: Find out what the remote machine's mounts are
      slurp:
        src: '{{ item }}'
      register: files
      loop:
        - ./files/example.json
        - ./files/op_content.json

    - debug:
        msg: "{{ item['content'] | b64decode }}"
      loop: '{{ files["results"] }}'

I am slurping each file, and then iterating through the results to get its content.

I hope it helps.