CODEWITHSUNDEEP
salt-project
HMITI
HMITI

Reputation: 11

SaltStack events got lost?

We setup a Salt state which will send an event to Salt Master when certain files missing. We also configured a reactor to capture the event and copy missing files accordingly. It is tested successfully on any single minion. However, when we apply the state to multiple minions (number range from 8 to 80+), only a portion (number range from 5 to 40+, randomly) of the minions triggered the reactors. We saw the events all being sent, just reactor never picks up all of them. With no configuration change, if we run the state again, some of the remaining minions (again, number is random) would trigger the reactor and get the missing files. Only after enough re-states, all of the minions will be completed successfully.

We tried to adjust REACTOR_WORKER_THREADS and/or REACTOR_WORKER_HWM, thinking maybe the queue is too small. However, I don't think it helped.

Upvotes: 1

Views: 132

Answers (1)

Megazord
Megazord

Reputation: 71

did you try to add the

  • queue=True

parameter on your state file, look below:

{% if data['tag'] == 'picachu' %}

trigger_picachu:
  local.state.apply:
    - tgt: {{ data['id'] }}
    - arg:
      - gitfs.picachu
      - queue=True
{% endif %}

Upvotes: 1

Related Questions