CODEWITHSUNDEEP
salt-project
Fernando Martin
Fernando Martin

Reputation: 689

3-level topology with SaltStack Salt Syndic performs very slowly

I've designed a 3-level infrastructure with 3 virtual machines (testfer-01, testfer-02 & testfer-03) and for the 3rd one is not possible to reach the 1st one so the 2nd must act as a middleman. To that end I've used the salt-syndic daemon in this second layer. More info: http://docs.saltstack.com/en/latest/topics/topology/syndic.html

To sum up, this is what is running on each VM:

testfer-01: salt-master
testfer-02: salt-master, salt-syndic & salt-minion
testfer-03: salt-minion

And I've set up the following files (every other option is set by default):

testfer-01:/etc/salt/master:

order_masters: True

testfer-02:/etc/salt/master:

syndic_master: testfer-01

testfer-02:/etc/salt/minion:

master: testfer-01

testfer-03:/etc/salt/minion:

master: testfer-02

Then, once the keys have been accepted I've "successfully" tested the connectivity from the top-layer machine. However, is taking too much time:

[root@testfer-01 ~]# time salt '*' test.ping
testfer-02.local:
    True
testfer-03.local:
    True

real    0m15.412s
user    0m0.716s
sys     0m0.216s

Anyone have any idea why?

EDIT: The command/module output is displayed (those "True" strings in this example) quite fast (no more than 2secs). But then, if I strace the process this one shows a lot of gettimeofday calls for around 10 more seconds...

...
poll([{fd=30, events=POLLIN}], 1, 0)    = 0 (Timeout)
gettimeofday({1421223627, 431608}, NULL) = 0
select(0, NULL, NULL, NULL, {0, 10000}) = 0 (Timeout)
poll([{fd=9, events=POLLIN}], 1, 0)     = 0 (Timeout)
gettimeofday({1421223627, 441964}, NULL) = 0
poll([{fd=30, events=POLLIN}], 1, 0)    = 0 (Timeout)
gettimeofday({1421223627, 442102}, NULL) = 0
select(0, NULL, NULL, NULL, {0, 10000}) = 0 (Timeout)
poll([{fd=9, events=POLLIN}], 1, 0)     = 0 (Timeout)
gettimeofday({1421223627, 452523}, NULL) = 0
poll([{fd=30, events=POLLIN}], 1, 0)    = 0 (Timeout)
gettimeofday({1421223627, 452720}, NULL) = 0
...

Upvotes: 2

Views: 1308

Answers (2)

Fernando Martin
Fernando Martin

Reputation: 689

With version 2016.3 and no other single modification:

[root@testfer-01 ~]# salt-master --version
salt-master 2016.3.1 (Boron)
[root@testfer-01 ~]# time salt '*' test.ping
testfer-02:
    True
testfer-03:
    True

real    0m5.744s
user    0m0.552s
sys     0m0.123s

Upvotes: 0

Utah_Dave
Utah_Dave

Reputation: 4581

We've just release version 2016.3 on repo.saltstack.com and this has several very important speed improvements and fixes for the syndic. Can you test against that?

Upvotes: 1

Related Questions