CODEWITHSUNDEEP
salt-project
Tomáš Fejfar
Tomáš Fejfar

Reputation: 11217

How to join two Salt pillar files and merge data?

Is there any way to join two pillar files?

I have a users pillar. It's something like:

users:
  joe:
    sudouser: True
  jack:
    sudouser: False

Now I need different set of users for certain servers (ie. add some users to one server). So I create new pillar file:

users:
  new_user:
    sudouser: True

And assign this topfile to the server. But because the key is the same it would overwrite the first one. If I change it I would need to update the state file (which I really don't want). How should I approach this problem? Is there any way to tell salt to "merge" the files?

Upvotes: 14

Views: 14520

Answers (4)

OrangeDog
OrangeDog

Reputation: 38777

As of Salt 2014.7.0, Salt will merge pillar keys by default. To control the behaviour, configure the pillar_source_merging_strategy and pillar_merge_lists options.

Upvotes: 1

perfecto25
perfecto25

Reputation: 842

The way I got around it is by changing the list values to a dict, for example

/srv/pillar/common/packages.sls

packages:
   htop: { pkg=installed }
   rsync: { pkg=removed }
   wget: { pkg=installed }



/srv/pillar/servers/nycweb01.sls
packages:
  nginx: { pkg=installed }

checking this servers pillar items you can see it combined the data from both the common pillar and per-node pillar,

salt-ssh nycweb01 pillar.items

nycweb01:
    ----------
    packages:
        ----------
        htop:
            ----------
            pkg=installed:
                None
        nginx:
            ----------
            pkg=installed:
                None
        rsync:
            ----------
            pkg=removed:
                None
        wget:
            ----------
            pkg=installed:

And from the state file, you can use both the pkg name and the pkg state(installed, removed, etc)

Upvotes: 0

uvsmtid
uvsmtid

Reputation: 4295

It is possible at least according to the latest Salt documentation about pillar (as of 5188d6c) which states:

With some care, the pillar namespace can merge content from multiple pillar files under a single key, so long as conflicts are avoided ...

I tested it under Salt Helium (2014.7.0) and it's working as expected.

Your Example

Pillar file user_set_a.sls:

users:
  joe:
    sudouser: True
  jack:
    sudouser: False

Pillar file user_set_b.sls:

users:
  new_user:
    sudouser: True

Run pillar.items to confirm that all users are merged under the same users key:

salt-call pillar.items
...
    users:
    ----------
    jack:
        ----------
        sudouser:
            False
    joe:
        ----------
        sudouser:
            True
    new_user:
        ----------
        sudouser:
            True
...

See also:

Upvotes: 13

mway
mway

Reputation: 4392

Short answer: you can't merge pillar data in this way.

Long answer: the pillar doesn't support the extend keyword the same way the state tree does, though there is some conversation on salt issue #3991. Unfortunately, there doesn't seem to be any real momentum with this at the moment and I'm not aware of any plans for this to be included in Helium.

Realistically, you'd be better off ensuring that your pillar data is distinct on a per-minion basis, and then you won't need to worry about collisions. You could optionally do something with YAML anchors and references, e.g.

# common/base users.sls
base_users: &base_users
    user1:
       foo: bar
    user2:
       baz: bat

# minion1.sls
{% include 'common/base_users.sls' %}

users:
    <<: *base_users
    user3:
        qux: quux

# minion2.sls
{% include 'common/base_users.sls' %}

users:
    <<: *base_users
    user4:
        corge: grault

Another potential (hacky) option is to use an external pillar module and do some sort of glob matching on pillar keys provided to the module, so you could basically have keys like merge-thing-abc123 and merge-thing-def456, using the merge prefix to group by thing and combine the data. I wouldn't really recommend this as it's a pretty blatant antipattern WRT pillar data (not to mention difficult to maintain).

For what it's worth, this is something that also frustrates me occasionally, but I end up deciding that some minimal data duplication is better than coming up with a workaround. Using the YAML references, this could potentially be a more agreeable option since technically you don't need to duplicate data, and is more easily maintainable. Granted, you end up polluting the pillar with extra unused keys (e.g. base_users), but in this particular case I'd consider that acceptable.

Hope this helps!

Edit: I may have spoke too soon; it looks as though includes are parsed prior to being injected into the including file, so anchors/references wouldn't work in that case. Looking into it, will update.

Edit 2: Just occurred to me that since both state and pillar files are essentially Python modules, they can be included with Jinja vs using pillar's include. So, instead of

include:
    - common.base_users

you can do

{% include 'common/base_users.sls' %}

and then proceed to reference any anchors defined in the included document. Updated the original answer to illustrate this (verified to work).

Upvotes: 2

Related Questions