Ansible rollback: run a group of tasks over list of hosts even when one of hosts failed

Question

I have a playbook with multiple roles, hosts and groups. I am trying to develop a rollback functionality, that would run over all hosts. My current obstacle is that I see no way to delegate role, block or set of tasks to group of hosts

• I tried looking up delegation to group without loops, so it would work on a block.
• import_role doesn't accept loops
• include_role doesn't accept delegate_to
• same with import_tasks/include_tasks

here is what I have now as a playbook file (shortened version)

- hosts: all
  any_errors_fatal: true
  vars_prompt:

  - name: "remote_user_p"
    prompt: "Remote user running the playbook"
    default: "root"
    private: no

  - name: "service_user_p"
    prompt: "Specify user to run non-root tasks"
    default: "user"
    private: no

  tasks:
    - set_fact:
        playbook_type: "upgrade"

    - import_role:
        name: 0_pre_check
      run_once: true
      remote_user: "{{ remote_user_p }}"
      become_user: "{{ service_user_p }}"
      become_method: su
      become: yes

    - block:      
      - import_role:
          name: 1_os

      - import_role:
          name: 2_mysql
        when: inventory_hostname in groups['mysql'] | default("")

      - import_role:
          name: 3_web
        when: inventory_hostname in groups['web'] | default("") 
...

      rescue:
        - block:
          - name: run rollback
            import_tasks: ../common/roles/5_rollback/tasks/rollback.yml

      remote_user: "{{ remote_user }}"
      become_user: "{{ service_user }}"
      become_method: su
      become: yes

This is some example code from rollback.yml:

- block:
  - name: rollback symlinks to config dir
    file:
      src: "{{ current_config_path }}"
      dest: "{{ install_dir }}/static/cfg"
      owner: "{{ service_user }}"
      group: "{{ service_user_primary_group }}"
      state: link
    when: current_new_configs | default("N") == "Y"
    delegate_to: "{{ item }}"
    with_items:
      - "{{ ansible_play_hosts }}"

  - block:           
    - name: return config files
      shell: test -f '{{ item.1.current_ver_file_path }}' && cp -p {{ item.1.current_ver_file_path }} {{ item.1.old_config_location }}
      args:
        warn: false
      register: return_config_files
      failed_when: return_config_files.rc >= 2
      when:
        - roolback_moved_cfg | default('N') == "Y"
        - inventory_hostname in groups[item.0.group]
        - item.1.old_config_location != ""
        - item.1.current_ver_file_path != ""
      with_subelements:
        - "{{ config_files }}"
        - files
      become_user: root
      become_method: sudo
      become: yes

    - name: systemctl daemon-reload  
      shell: systemctl daemon-reload
      failed_when: false
      when: root_rights == "Y"
      args:
        warn: false
      delegate_to: "{{ item }}"
      with_items:
        - "{{ ansible_play_hosts }}"
    when: root_rights == "Y"
    become_user: root
    become_method: sudo
    become: yes

  - fail:
      msg: "Upgrade failed. Symbolic links were set to the previous version. Fix the issues and try again. If you wish to cancel the upgrade, restore the database backup manually."

As you can see, now I use lame workaround by introducing

      delegate_to: "{{ item }}"
      with_items:
        - "{{ ansible_play_hosts }}"

after every task.

There are two problems here: 1. I can't use same approach after task return config files, because it already uses one loop 2. This is generally lame duplication of code and I hate it

Why I need it at all: if playbook execution fails somewhere in mysql role, for example, the rescue block will be executed only over the hosts in that mysql role (and btw, execution of tasks from next role will continue while running rescue block - same amount of tasks, despite all efforts), while I would like it to run over all hosts instead.

Answer 1

include_role doesn't accept delegate_to

Actually, it does.

With ansible-core 2.8:

- name: "call my/role with host '{{ansible_hostname}}' for hosts in '{{ansible_play_hosts}}'"
  include_role:
    name: my/role
  apply:
    delegate_to: "{{current_host}}"
  with_items: "{{ansible_play_hosts}}"
    loop_control:
      loop_var: current_host

With ansible-core 2.5 to 2.7, see "2.5: delegate_to, include_role with loops" from George Shuklin, mentioned in ansible/ansible issue 35398

- name: "call my/role with host '{{ansible_hostname}}' for items in '{{ansible_play_hosts}}'"
    include_tasks: loop.yml
    with_items: "{{ansible_play_hosts}}"
    loop_control:
      loop_var: current_host

With loop.yml another tasks in its own file:

- name: "Import my/role for '{{current_host}}'"
  import_role: name=my/role
  delegate_to: "{{current_host}}"

So in two files (with ansible-core 2.7) or one file (2.8), you can make a all role and its tasks run on a delegated server.

Answer 2

I finally was able to solve this with an ugly-ugly hack. Used plays instead of just roles - now there are more than 10 plays. Don't judge me, I spent lots of effort trying to make it nice ):

Example play followed by a check - same as for every other.

- hosts: mysql
  any_errors_fatal: true
  tasks:
    - block:              
      - import_role:
          name: 2_mysql
        when: not rollback | default(false)
      rescue:
        - block:
          - name: set fact for rollback
            set_fact: 
              rollback: "yes"
            delegate_to: "{{ item }}"
            delegate_facts: true
            with_items: "{{ groups['all'] }}"

- hosts: all
  any_errors_fatal: true
  tasks:
    - name: run rollback
      import_tasks: ../common/roles/5_rollback/tasks/rollback.yml
      when: rollback | default(false)