puppet: Hiera does not like any sub-directories

Question

Can any Puppet and Hiera experts out there help me out with this annoying situation I'm seeing?

Hiera works fine for anything in common.yaml:

cat common.yaml
---
resolv_conf::nameserver:
- x.x.x.x
- x.x.x.x

profile::ntp::servers:
- x.x.x.x
- x.x.x.x

Then tested with the following works:

hiera resolv_conf::nameservers environment=production
["x.x.x.x", "x.x.x.x"]

So, the above works as expected. However, if I use a nodes directory inside hieradata and add test data in:

cat nodes/fqdn.yaml
---
profile::ntp::servers:
-1.1.1.1
-2.2.2.2

Then I adjust my hiera.yaml file accordingly:

cat hiera.yaml
---
:backends: yaml
:yaml:
  :datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"
:hierarchy:
  -"nodes/%{::fqdn}"
  - "common"

:logger: console
:merge_behavior: native
:deep_merge_options: {}

And also add the correct line into the global hiera.yaml (both files now look the same for testing).

When testing I get the following error:

hiera profile::ntp::servers environment=production -d

DEBUG: xxxxxxxxx: Hiera YAML backend starting
DEBUG: xxxxxxxxx: Looking up profile::ntp::servers in YAML backend
DEBUG: xxxxxxxxx: Ignoring bad definition in :hierarchy: 'nodes/'
DEBUG: xxxxxxxxx: Looking for data source common
DEBUG: xxxxxxxxx: Found profile::ntp::servers in common
["x.x.x.x", "x.x.x.x"]

So it finds the values in common but doesn't like the nodes directory.

I have tried loads of different variations of the hierarchy line in hiera.yaml, and many different options everywhere else I can imagine!

There may be typos in the above, as I had to type it all out manually as I have no copy and paste, but I have checked the live files again and again for typos and syntax errors.

Any ideas?

Some updates now I'm home and can copy/paste:

This is on a fresh install of puppet enterprise, using code manager and gitlab.

[root@client-01 nodes]# pwd
/root/control-repo/hieradata/nodes
[root@client-01 nodes]# ll
total 8
-rw-r--r--. 1 root root 34 Jul 13 19:04 puppet.example.com.yaml
-rw-r--r--. 1 root root 18 Jul 13 18:58 test.yaml
[root@client-01 nodes]# cat puppet.example.com.yaml
---
message:
- "This is in nodes"

This is on a dev machine, the repo was then pushed which initiated the hook for code deploy, I've checked that the same data is available in /etc/puppetlabs/code/environments/production etc on the puppet master. So Code Manager is working.

The following tests on the puppet server:

[root@puppet ~]# hostname -f
puppet.example.com
[root@puppet ~]# hiera message environment=production
nil
[root@puppet ~]# hiera message environment=production 
fqdn=puppet.example.com
nil
[root@puppet ~]# hiera message environment=production fqdn=puppet.example.com -d
DEBUG: 2017-07-13 19:28:15 +0100: Hiera YAML backend starting
DEBUG: 2017-07-13 19:28:15 +0100: Looking up message in YAML backend
DEBUG: 2017-07-13 19:28:15 +0100: Ignoring bad definition in :hierarchy: 'nodes/'
DEBUG: 2017-07-13 19:28:15 +0100: Looking for data source common
nil
[root@puppet ~]#

And so it looks like exactly the same problem:

My hiera.yaml:

[root@puppet ~]# cat /etc/puppetlabs/puppet/hiera.yaml
---
:backends:
  - yaml
:hierarchy:
  - "nodes/%{::fqdn}"
  - common

:yaml:
# datadir is empty here, so hiera uses its defaults:
# - /etc/puppetlabs/code/environments/%{environment}/hieradata on *nix
# - %CommonAppData%\PuppetLabs\code\environments\%{environment}\hieradata on Windows
# When specifying a datadir, make sure the directory exists.
  :datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"

This is somewhat reassuring as it means maybe there isn't something majorly wrong with the other puppet server and more than likely something I'm doing wrong?

FURTHER EDIT:

So I didn't have too much time last night to test things but I did find this:

It looks like the problem is with the facter variables i.e. `"%{::fqdn}"

If I do the following in the root of hieradata:

[root@client-01 hieradata]# pwd ; ll ; cat test.yaml
/root/control-repo/hieradata
total 12
-rw-r--r--. 1 root root  4 Jul 13 19:25 common.yaml
drwxr-xr-x. 2 root root 52 Jul 13 19:04 nodes
-rw-r--r--. 1 root root 19 Jul 13 21:47 puppet.example.com.yaml
-rw-r--r--. 1 root root 18 Jul 13 21:43 test.yaml
---
name:
- "jon"

Then test with hiera name environment=production it works fine.

If I do the following in the root of hieradata:

[root@client-01 hieradata]# pwd ; ll ; cat puppet.example.com.yaml
/root/control-repo/hieradata
total 12
-rw-r--r--. 1 root root  4 Jul 13 19:25 common.yaml
drwxr-xr-x. 2 root root 52 Jul 13 19:04 nodes
-rw-r--r--. 1 root root 19 Jul 13 21:47 puppet.example.com.yaml
-rw-r--r--. 1 root root 18 Jul 13 21:43 test.yaml
---
test:
- "pass"

Then I test in the following ways:

[root@puppet ~]# hostname -f ; hiera test environment=production -d
puppet.example.com
DEBUG: 2017-07-13 21:53:44 +0100: Hiera YAML backend starting
DEBUG: 2017-07-13 21:53:44 +0100: Looking up test in YAML backend
DEBUG: 2017-07-13 21:53:44 +0100: Ignoring bad definition in :hierarchy: 
'nodes/'
DEBUG: 2017-07-13 21:53:44 +0100: Ignoring bad definition in :hierarchy: ''
DEBUG: 2017-07-13 21:53:44 +0100: Looking for data source test
DEBUG: 2017-07-13 21:53:44 +0100: Looking for data source common
nil
[root@puppet ~]# hostname -f ; hiera test environment=production 
fqdn=puppet.example.com -d
puppet.example.com
DEBUG: 2017-07-13 21:53:57 +0100: Hiera YAML backend starting
DEBUG: 2017-07-13 21:53:57 +0100: Looking up test in YAML backend
DEBUG: 2017-07-13 21:53:57 +0100: Ignoring bad definition in :hierarchy: 
'nodes/'
DEBUG: 2017-07-13 21:53:57 +0100: Ignoring bad definition in :hierarchy: ''
DEBUG: 2017-07-13 21:53:57 +0100: Looking for data source test
DEBUG: 2017-07-13 21:53:57 +0100: Looking for data source common
nil

So disregarding the subfolder of nodes entirely and testing in the root of hieradata shows that yaml files with a fixed name work, but when using facter for a per node lookup... this is where the failure is.

[root@puppet ~]# facter fqdn
puppet.example.com
[root@puppet ~]# hostname -f
puppet.example.com

However, facter seems to be working just fine.

Any help would be greatly appreciated, thanks. Jon

Answer 1

I reproduced this and the issue is in fact that the scope indicators need to be included in the facts on the hiera command line.

I.e. instead of:

# hiera test environment=production fqdn=puppet.example.com -d

you should say:

# hiera test environment=production ::fqdn=puppet.example.com -d

To reproduce in Rspec-puppet:

$ cat manifests/init.pp 
class foo (
  $servers,
  $nameserver,
) {
  notify { "servers: $servers": }
  notify { "nameserver: $nameserver": }
}

And

$ cat spec/fixtures/hiera/hiera.yaml 
---
:backends:
  - yaml
:hierarchy:
  - "nodes/%{::fqdn}"
  - common
:yaml:
  :datadir: 'spec/fixtures/hieradata'
:logger: console
:merge_behavior: native
:deep_merge_options: {}

And

$ cat spec/fixtures/hieradata/common.yaml 
---
foo::nameserver:
- 3.3.3.3
- 4.4.4.4

foo::servers:
- 5.5.5.5
- 6.6.6.6

And

$ cat spec/fixtures/hieradata/nodes/myhost.example.com.yaml 
---
foo::servers:
- 1.1.1.1
- 2.2.2.2

And

$ cat spec/spec_helper.rb 
require 'puppetlabs_spec_helper/module_spec_helper'

RSpec.configure do |c|
  c.hiera_config = 'spec/fixtures/hiera/hiera.yaml'
  c.default_facts = {
    :fqdn => 'myhost.example.com',
  }
end

And

$ cat spec/classes/test_spec.rb 
require 'spec_helper'

describe 'foo', :type => :class do
  it { File.write('myclass.json', PSON.pretty_generate(catalogue)) }
end

Compile:

$ bundle exec rake spec 
...
Finished in 0.15682 seconds (files took 0.97868 seconds to load)
1 example, 0 failures

Apply that:

$ bundle exec puppet apply --environment=rp_env --catalog=myclass.json 
Notice: servers: [1.1.1.1, 2.2.2.2]
Notice: /Stage[main]/Foo/Notify[servers: [1.1.1.1, 2.2.2.2]]/message: defined 'message' as 'servers: [1.1.1.1, 2.2.2.2]'
Notice: nameserver: [3.3.3.3, 4.4.4.4]
Notice: /Stage[main]/Foo/Notify[nameserver: [3.3.3.3, 4.4.4.4]]/message: defined 'message' as 'nameserver: [3.3.3.3, 4.4.4.4]'
Notice: Applied catalog in 0.02 seconds

I change the value of fqdn in spec_helper and compile and apply again:

$ bundle exec puppet apply --environment=rp_env --catalog=myclass.json 
Notice: servers: [5.5.5.5, 6.6.6.6]
Notice: /Stage[main]/Foo/Notify[servers: [5.5.5.5, 6.6.6.6]]/message: defined 'message' as 'servers: [5.5.5.5, 6.6.6.6]'
Notice: nameserver: [3.3.3.3, 4.4.4.4]
Notice: /Stage[main]/Foo/Notify[nameserver: [3.3.3.3, 4.4.4.4]]/message: defined 'message' as 'nameserver: [3.3.3.3, 4.4.4.4]'
Notice: Applied catalog in 0.01 seconds

(rp_env is the magic value for the environment that Rspec-puppet sets.)

So the code works.

But it's the Hiera command line that has issues.

Reproducing your error message:

$ bundle exec hiera foo::servers environment=rp_env -d -c spec/fixtures/hiera/hiera.yaml 
DEBUG: 2017-07-15 13:21:54 +1000: Hiera YAML backend starting
DEBUG: 2017-07-15 13:21:54 +1000: Looking up foo::servers in YAML backend
DEBUG: 2017-07-15 13:21:54 +1000: Ignoring bad definition in :hierarchy: 'nodes/'
DEBUG: 2017-07-15 13:21:54 +1000: Looking for data source common
DEBUG: 2017-07-15 13:21:54 +1000: Found foo::servers in common
["5.5.5.5", "6.6.6.6"]

Attempt to set the fact on the command line using the same syntax you did:

$ bundle exec hiera foo::servers fqdn=myhost.example.com environment=rp_env -d -c spec/fixtures/hiera/hiera.yaml 
DEBUG: 2017-07-15 13:23:34 +1000: Hiera YAML backend starting
DEBUG: 2017-07-15 13:23:34 +1000: Looking up foo::servers in YAML backend
DEBUG: 2017-07-15 13:23:34 +1000: Ignoring bad definition in :hierarchy: 'nodes/'
DEBUG: 2017-07-15 13:23:34 +1000: Looking for data source common
DEBUG: 2017-07-15 13:23:34 +1000: Found foo::servers in common
["5.5.5.5", "6.6.6.6"]

Finally providing the missing scope indicator:

$ bundle exec hiera foo::servers ::fqdn=myhost.example.com environment=rp_env -d -c spec/fixtures/hiera/hiera.yaml 
DEBUG: 2017-07-15 13:24:09 +1000: Hiera YAML backend starting
DEBUG: 2017-07-15 13:24:09 +1000: Looking up foo::servers in YAML backend
DEBUG: 2017-07-15 13:24:09 +1000: Looking for data source nodes/myhost.example.com
DEBUG: 2017-07-15 13:24:09 +1000: Found foo::servers in nodes/myhost.example.com
["1.1.1.1", "2.2.2.2"]

Hooray it works.

Agree that's pretty confusing.

See also here.

Answer 2

The key information from the debug output is:

DEBUG: xxxxxxxxx: Ignoring bad definition in :hierarchy: 'nodes/'

This should be an indication that your YAML hieradata for that lookup priority has a syntax error. Sure enough, if we use an online validation tool like http://www.yamllint.com, we see it has a syntax error. Change your hieradata file to:

# nodes/myhost.example.com.yaml
---
profile::ntp::servers:
- 1.1.1.1
- 2.2.2.2

and it should work fine.

I recommend in the future to use a tool or some code to validate your code and data so you can identify these issues faster and easier. I have written a popular one for that, but will refrain from plugging it as I think StackOverflow may not be happy about that.

Answer 3

In your hiera.yaml you have:

-"nodes/%{::fqdn}"

In there, %{::fqdn} refers to the Facter fact fqdn.

Meanwhile you created a file called nodes/fqdn.yaml.

Hiera would not see that file unless your fqdn fact was literally returning fqdn (rather than something like myhost.example.com).

It should find the file if you instead say:

hiera profile::ntp::servers environment=production fqdn=fqdn -d

However what you probably want to do is rename the Hiera file/s to match your nodes' actual FQDNs, i.e.

cat nodes/myhost.example.com.yaml
---
profile::ntp::servers:
-1.1.1.1
-2.2.2.2