CODEWITHSUNDEEP
ansiblebatchsize
Mk_head
Mk_head

Reputation: 23

Controlling Ansible playbook execution

Is it possible to control batch execution in percentage, but applying it individually to groups of hosts in the Ansible playbook?

my hosts file:

group_1 - has 5 hosts
group_2 - has 18 hosts
group_3 - has 4 hosts

And my playbook like this:

- name: "Apply percent per group"
  hosts: all
  serial: "30%"

Currently, when I run like this, Ansible concatenate all groups and applies the 30% serial, so it runs 8 hosts per round and linearly, so it ends up getting all the hosts in group_1 at once.

Is there a way to apply this 30% per group individually?

So per batch it would run 1 host from group_1, 5 hosts from group_2 and 1 host from group_3.

It would even give me the possibility to increase the percentage that would still not cause unavailability and decrease execution time.

Upvotes: 2

Views: 1005

Answers (2)

Vladimir Botka
Vladimir Botka

Reputation: 68144

Create the batch on your own. For example, given the inventory

shell> cat hosts
[group_1]
host_1_[01:05]
[group_2]
host_2_[01:18]
[group_3]
host_3_[01:04]

and the variable my_serial_pc

  my_serial_pc: 30
  1. Declare the variables
  count_1: "{{ groups.group_1|length * my_serial_pc|int // 100 }}"
  count_2: "{{ groups.group_2|length * my_serial_pc|int // 100 }}"
  count_3: "{{ groups.group_3|length * my_serial_pc|int // 100 }}"
  batch_1: "{{ groups.group_1|batch(count_1|int) }}"
  batch_2: "{{ groups.group_2|batch(count_2|int) }}"
  batch_3: "{{ groups.group_3|batch(count_3|int) }}"
  my_serial: "{{ count_1|int + count_2|int + count_3|int }}"

give

  count_1: 1
  count_2: 5
  count_3: 1
  batch_1:
    - [host_1_01]
    - [host_1_02]
    - [host_1_03]
    - [host_1_04]
    - [host_1_05]
  
  batch_2:
    - [host_2_01, host_2_02, host_2_03, host_2_04, host_2_05]
    - [host_2_06, host_2_07, host_2_08, host_2_09, host_2_10]
    - [host_2_11, host_2_12, host_2_13, host_2_14, host_2_15]
    - [host_2_16, host_2_17, host_2_18]
 
  batch_3:
    - [host_3_01]
    - [host_3_02]
    - [host_3_03]
    - [host_3_04]

  my_serial: 7
  1. zip and flatten the batches
  my_hosts: "{{ batch_1|
                zip_longest(batch_2)|map('flatten')|
                zip_longest(batch_3)|map('flatten')|flatten }}"
  1. Create a new group my_group
        - add_host:
            name: "{{ item }}"
            groups: my_group
            my_serial: "{{ my_serial }}"
          loop: "{{ my_hosts }}"
  1. Run the next play with the group my_group and serial my_serial
- hosts: my_group
  serial: "{{ hostvars[groups.my_group.0].my_serial }}"
  tasks:
    - debug:
        var: ansible_play_batch|to_yaml
      run_once: true

gives

PLAY [my_group] *******************************************************************************************

TASK [debug] **********************************************************************************************
ok: [host_1_01] => 
  ansible_play_batch|to_yaml: |-
    [host_1_01, host_2_01, host_2_02, host_2_03, host_2_04, host_2_05, host_3_01]

PLAY [my_group] *******************************************************************************************

TASK [debug] **********************************************************************************************
ok: [host_1_02] => 
  ansible_play_batch|to_yaml: |-
    [host_1_02, host_2_06, host_2_07, host_2_08, host_2_09, host_2_10, host_3_02]

PLAY [my_group] *******************************************************************************************

TASK [debug] **********************************************************************************************
ok: [host_1_03] => 
  ansible_play_batch|to_yaml: |-
    [host_1_03, host_2_11, host_2_12, host_2_13, host_2_14, host_2_15, host_3_03]

PLAY [my_group] *******************************************************************************************

TASK [debug] **********************************************************************************************
ok: [host_1_04] => 
  ansible_play_batch|to_yaml: |-
    [host_1_04, host_2_16, host_2_17, host_2_18, host_3_04, host_1_05]

Example of a complete playbook for testing

shell> cat pb.yml
- hosts: all

  vars:

    my_serial_pc: 30
    count_1: "{{ groups.group_1|length * my_serial_pc|int // 100 }}"
    count_2: "{{ groups.group_2|length * my_serial_pc|int // 100 }}"
    count_3: "{{ groups.group_3|length * my_serial_pc|int // 100 }}"
    batch_1: "{{ groups.group_1|batch(count_1|int) }}"
    batch_2: "{{ groups.group_2|batch(count_2|int) }}"
    batch_3: "{{ groups.group_3|batch(count_3|int) }}"
    my_hosts: "{{ batch_1|
                  zip_longest(batch_2)|map('flatten')|
                  zip_longest(batch_3)|map('flatten')|flatten }}"
    my_serial: "{{ count_1|int + count_2|int + count_3|int }}"

  tasks:

    - block:
        - debug:
            msg: |
              count_1: {{ count_1 }}
              count_2: {{ count_2 }}
              count_3: {{ count_3 }}
              batch_1:
                {{ batch_1|to_yaml|indent(2) }}
              batch_2:
                {{ batch_2|to_yaml|indent(2) }}
              batch_3:
                {{ batch_3|to_yaml|indent(2) }}
              my_serial: {{ my_serial }}
          when: debug|d(false)|bool
        - add_host:
            name: "{{ item }}"
            groups: my_group
            my_serial: "{{ my_serial }}"
          loop: "{{ my_hosts }}"
      run_once: true

- hosts: my_group
  serial: "{{ hostvars[groups.my_group.0].my_serial }}"
  tasks:
    - debug:
        var: ansible_play_batch|to_yaml
      run_once: true

gives

shell> ansible-playbook pb.yml -e debug=true

PLAY [all] ************************************************************************************************

TASK [debug] **********************************************************************************************
ok: [host_1_01] => 
  msg: |-
    count_1: 1
    count_2: 5
    count_3: 1
    batch_1:
      - [host_1_01]
      - [host_1_02]
      - [host_1_03]
      - [host_1_04]
      - [host_1_05]
  
    batch_2:
      - [host_2_01, host_2_02, host_2_03, host_2_04, host_2_05]
      - [host_2_06, host_2_07, host_2_08, host_2_09, host_2_10]
      - [host_2_11, host_2_12, host_2_13, host_2_14, host_2_15]
      - [host_2_16, host_2_17, host_2_18]
  
    batch_3:
      - [host_3_01]
      - [host_3_02]
      - [host_3_03]
      - [host_3_04]
  
    my_serial: 7

TASK [add_host] *******************************************************************************************
changed: [host_1_01] => (item=host_1_01)
changed: [host_1_01] => (item=host_2_01)
changed: [host_1_01] => (item=host_2_02)
changed: [host_1_01] => (item=host_2_03)
changed: [host_1_01] => (item=host_2_04)
changed: [host_1_01] => (item=host_2_05)
changed: [host_1_01] => (item=host_3_01)
changed: [host_1_01] => (item=host_1_02)
changed: [host_1_01] => (item=host_2_06)
changed: [host_1_01] => (item=host_2_07)
changed: [host_1_01] => (item=host_2_08)
changed: [host_1_01] => (item=host_2_09)
changed: [host_1_01] => (item=host_2_10)
changed: [host_1_01] => (item=host_3_02)
changed: [host_1_01] => (item=host_1_03)
changed: [host_1_01] => (item=host_2_11)
changed: [host_1_01] => (item=host_2_12)
changed: [host_1_01] => (item=host_2_13)
changed: [host_1_01] => (item=host_2_14)
changed: [host_1_01] => (item=host_2_15)
changed: [host_1_01] => (item=host_3_03)
changed: [host_1_01] => (item=host_1_04)
changed: [host_1_01] => (item=host_2_16)
changed: [host_1_01] => (item=host_2_17)
changed: [host_1_01] => (item=host_2_18)
changed: [host_1_01] => (item=host_3_04)
changed: [host_1_01] => (item=host_1_05)

PLAY [my_group] *******************************************************************************************

TASK [debug] **********************************************************************************************
ok: [host_1_01] => 
  ansible_play_batch|to_yaml: |-
    [host_1_01, host_2_01, host_2_02, host_2_03, host_2_04, host_2_05, host_3_01]

PLAY [my_group] *******************************************************************************************

TASK [debug] **********************************************************************************************
ok: [host_1_02] => 
  ansible_play_batch|to_yaml: |-
    [host_1_02, host_2_06, host_2_07, host_2_08, host_2_09, host_2_10, host_3_02]

PLAY [my_group] *******************************************************************************************

TASK [debug] **********************************************************************************************
ok: [host_1_03] => 
  ansible_play_batch|to_yaml: |-
    [host_1_03, host_2_11, host_2_12, host_2_13, host_2_14, host_2_15, host_3_03]

PLAY [my_group] *******************************************************************************************

TASK [debug] **********************************************************************************************
ok: [host_1_04] => 
  ansible_play_batch|to_yaml: |-
    [host_1_04, host_2_16, host_2_17, host_2_18, host_3_04, host_1_05]

PLAY RECAP ************************************************************************************************
host_1_01: ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
host_1_02: ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
host_1_03: ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
host_1_04: ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Upvotes: 1

U880D
U880D

Reputation: 12121

According Setting the batch size with serial your observation is the expected behavior.

Ansible runs in parallel against all the hosts in the pattern you set in the hosts: field of each play.

So regarding

Is there a way to apply this 30% per group individually?

that means you would need to introducde additionally Patterns: targeting hosts and groups and set a limit to a group.

An other approach might be to implement a rolling update strategy whereby serial would have

... a list that specifies how to slowly increase the number of parallel processes or hosts to execute against. This play will execute on one host, then move to two hosts, then 25% of the hosts ...

See also in examlpe Running a playbook on multiple host groups one at a time.

You could also have more than one hosts section within your playbook, whereby each would start a new play. See i.e. How to specify different hosts for different playbooks in one Ansible script?.

Upvotes: 0

Related Questions