Ansible: how to achieve idempotence with tasks that append files on host (w/o reverting to initial state)

Question

I am having a hard time getting to know how to create Ansible roles that are following the best practices according to documentation. The following use-case which I am looking at is e.g. enabling Filebeat on host. Filebeat can be configured by placing a module definition in /etc/filebeat/modules.d folder.

It works fine when I am adding modules. Idempotence is working, everytime, on each run of the role (playbook), a given set of modules is enabled.

But what I should do when I decide that a given module is not longer needed? I remove it from role, rerun a playbook, so that all other modules are enabled. But: the previous run enabled a module that I am not installing directly with role after changes. So my server state is still altered in a way that is different than the role is imposing itself.

My question is: should I take care of removing modules before I apply them so I always start from, let's say, fresh state?

E.g.:

- name: Remove modules
  file:
    dest: "/etc/filebeat/modules.d/{{ item }}"
    state: absent
  loop:
    - "module1.yml"
    - "module2.yml"
    - "module3.yml" # It was being installed in previous role, but not now

- name: Enable modules via 'modules.d' directory
  template:
    src: "modules.d/{{ item }}"
    dest: "/etc/filebeat/modules.d/{{ item }}"
    mode: '0644'
  loop:
    - "module1.yml"
    - "module2.yml"

So I remove module3.yml, because I remember that I've installed it before, and install module1.yml and module2.yml.

Instead of just installing what I need, no matter what has been installed before:

- name: Enable modules via 'modules.d' directory
  template:
    src: "modules.d/{{ item }}"
    dest: "/etc/filebeat/modules.d/{{ item }}"
    mode: '0644'
  loop:
    - "module1.yml"
    - "module2.yml"

Leaving me with module1.yml and module2.yml (desired) and, unfortunately: module3.yml (from previous role).

How to manage that to avoid such situations? And avoid treating server as one big stateful machine that even if I run a role, the output is different than desired, because something has been done before that I cannot see in current Ansible role code.

Do you code revert playbooks in your Ansible workflow to revert to initial state when needed?

I am curious. Thanks in advance for your reply.

Answer 1

Remove all modules not listed if you want to stay safe.

Short answer: Look at the block 'Remove modules' in the example below the line.

Details: Given the directory on the remote host for testing

shell> ssh admin@test_11 ls -1 /tmp/etc/filebeat/modules.d
module99.yml

Declare the path to the modules and the list of the modules you want to configure

  fb_modules_path: /tmp/etc/filebeat/modules.d
  fb_modules:
    - module1.yml
    - module2.yml
    - module3.yml

Create the templates for the modules

tree> tree modules.d/
modules.d/
├── module1.yml.j2
├── module2.yml.j2
└── module3.yml.j2

and create the modules for a single remote host test_11 in this example

    - name: Create modules
      template:
        src: "modules.d/{{ item }}.j2"
        dest: "{{ fb_modules_path }}/{{ item }}"
        mode: '0644'
      loop: "{{ fb_modules }}"
      notify: restart_filebeat

PLAY [test_11] *******************************************************************************

TASK [Create modules] ************************************************************************
changed: [test_11] => (item=module1.yml)
changed: [test_11] => (item=module2.yml)
changed: [test_11] => (item=module3.yml)

To remove all modules that are not in the list fb_modules find all files in the directory fb_modules_path

        - name: Find all files in {{ fb_modules_path }}
          find:
            path: "{{ fb_modules_path }}"
          register: find_fb_modules

Declare the list of found modules and the list of the modules that should be removed

  fb_modules_found: "{{ find_fb_modules.files|map(attribute='path')|
                                              map('basename') }}"
  fb_modules_remove: "{{ fb_modules_found|difference(fb_modules) }}"

give

  fb_modules_found: ['module99.yml', 'module1.yml', 'module3.yml', 'module2.yml']
  fb_modules_remove: ['module99.yml']

Remove the modules

        - name: Remove modules
          file:
            dest: "{{ fb_modules_path }}/{{ item }}"
            state: absent
          loop: "{{ fb_modules_remove }}"
          notify: restart_filebeat

TASK [Remove modules] ************************************************************************
changed: [test_11] => (item=module99.yml)

Take a look at the result

shell> ssh admin@test_11 ls -1 /tmp/etc/filebeat/modules.d
module1.yml
module2.yml
module3.yml

---

Example of a complete playbook for testing

- hosts: test_11

  vars:

    fb_modules_path: /tmp/etc/filebeat/modules.d
    fb_modules:
      - module1.yml
      - module2.yml
      - module3.yml

    fb_modules_found: "{{ find_fb_modules.files|map(attribute='path')|
                                                map('basename') }}"
    fb_modules_remove: "{{ fb_modules_found|difference(fb_modules) }}"

  tasks:

    - name: Create modules
      template:
        src: "modules.d/{{ item }}.j2"
        dest: "{{ fb_modules_path }}/{{ item }}"
        mode: '0644'
      loop: "{{ fb_modules }}"
      notify: restart_filebeat

    - name: Remove modules
      block:
        - name: Find all files in {{ fb_modules_path }}
          find:
            path: "{{ fb_modules_path }}"
          register: find_fb_modules
        - debug:
            msg: |
              fb_modules_found: {{ fb_modules_found }}
              fb_modules_remove: {{ fb_modules_remove }}
          when: debug|d(false)|bool
        - name: Remove modules
          file:
            dest: "{{ fb_modules_path }}/{{ item }}"
            state: absent
          loop: "{{ fb_modules_remove }}"
          notify: restart_filebeat

  handlers:

    - name: Restart filebeat service
      listen: restart_filebeat
#     systemd:
#       name: filebeat
#       state: restarted
      debug:
        msg: Restart filebeat

Answer 2

In a nutshell:

- name: Configure filebeat modules
  hosts: all

  vars:
    fb_modules_d:
      - file: module1.yml
        state: present
      - file: module2.yml
        state: present
      - file: module3.yml
        state: absent

  tasks:
    - name: Make sure all needed module files are present
      template:
        src: "modules.d/{{ item.file }}"
        dest: "/etc/filebeat/modules.d/{{ item.file }}"
        mode: '0644'
      loop: "{{ fb_modules_d | selectattr('state', '==', 'present') }}"
      notifiy: restart_filebeat
 
    - name: Make sure all disabled modules are removed
      file:
        dest: "/etc/filebeat/modules.d/{{ item.file }}"
        state: "{{ item.state }}"
      loop: loop: "{{ fb_modules_d | selectattr('state', '==', 'absent') }}"
      notify: restart_filebeat

  handlers:
    - name: Restart filebeat service
      listen: restart_filebeat
      systemd:
        name: filebeat
        state: restarted

Note: I declared the variable inside the playbook for the example but that one one should most probably go inside your inventory (group or host level), and certainly not in a role (except in defaults for documentation)