Append a hash to a series of nested hashes in Puppet4

Question

Given is an hash of hashes in hieradata:

profile::jdbc::connections
  connection_name1:
    username: 'user1'
    password: 'pass1'
  connection_name2:
    username: 'user2'
    password: 'pass2'

and an hash of defaults in puppet code:

$jdbc_default = {  
  'testWhileIdle'                => true,
  'testOnBorrow'                 => true,
  'testOnReturn'                 => false,
  'timeBetweenEvictionRunsMillis'=> '30000',
  'maxActive'                    => '20',
  'maxWait'                      => '10000',
  'initialSize'                  => '5',
  'removeAbandonedTimeout'       => '600',
  'removeAbandoned'              => false,
  'logAbandoned'                 => true,
  'minEvictableIdleTimeMillis'   => '30001',
}

How can I add the defaults to each Hash in the connections hash?

Result can also be an array of hashes but a hash with the same keys as in the connection hash would be nice.

Answer 1

Puppet 4 provides a number of iteration functions that can be used here, but the clearest and easiest solution to understand is probably to use Puppet's map and merge functions (ref and ref):

  $connections = {
    'connection_name1' => {
      'username' => 'user1',
      'password' => 'pass1',
    },
    'connection_name2' => {
      'username' => 'user2',
      'password' => 'pass2',
    },
  }

  $jdbc_default = {
    'testWhileIdle'                => true,
    'testOnBorrow'                 => true,
    'testOnReturn'                 => false,
    'timeBetweenEvictionRunsMillis'=> '30000',
    'maxActive'                    => '20',
    'maxWait'                      => '10000',
    'initialSize'                  => '5',
    'removeAbandonedTimeout'       => '600',
    'removeAbandoned'              => false,
    'logAbandoned'                 => true,
    'minEvictableIdleTimeMillis'   => '30001',
  }

  $merged = $connections.map |$k,$v| {
    {$k => merge($jdbc_default, $v)}
  }

  notice($merged)

Then check it:

Notice: Scope(Class[main]): [{connection_name1 => {username => user1, password => pass1, testWhileIdle => true, testOnBorrow => true, testOnReturn => false, timeBetweenEvictionRunsMillis => 30000, maxActive => 20, maxWait => 10000, initialSize => 5, removeAbandonedTimeout => 600, removeAbandoned => false, logAbandoned => true, minEvictableIdleTimeMillis => 30001}}, {connection_name2 => {username => user2, password => pass2, testWhileIdle => true, testOnBorrow => true, testOnReturn => false, timeBetweenEvictionRunsMillis => 30000, maxActive => 20, maxWait => 10000, initialSize => 5, removeAbandonedTimeout => 600, removeAbandoned => false, logAbandoned => true, minEvictableIdleTimeMillis => 30001}}]
Notice: Compiled catalog for alexs-macbook-pro.local in environment production in 0.07 seconds
Notice: Applied catalog in 0.01 seconds

However, you mentioned that your data comes from Hiera. Therefore, your actual code would look like:

class profile::jdbc (
  Hash[String, Hash[String, String]] $connections,
) {
  $jdbc_default = {
    'testWhileIdle'                => true,
    'testOnBorrow'                 => true,
    'testOnReturn'                 => false,
    'timeBetweenEvictionRunsMillis'=> '30000',
    'maxActive'                    => '20',
    'maxWait'                      => '10000',
    'initialSize'                  => '5',
    'removeAbandonedTimeout'       => '600',
    'removeAbandoned'              => false,
    'logAbandoned'                 => true,
    'minEvictableIdleTimeMillis'   => '30001',
  }

  $merged = $connections.map |$k,$v| {
    {$k => merge($jdbc_default, $v)}
  }

  notice($merged)
}

Note that because Hashes can be added in Puppet, the merge function, which comes from stdlib, can be avoided:

  $merged = $connections.map |$k,$v| {
    {$k => $jdbc_default + $v}
  }

(Note that {'a' => 1} + {'b' => 2} returns {'a' => 1, 'b' => 2}. If keys are in both, the right-hand side wins, i.e. {'a' => 1, 'b' => 2} + {'a' => 2} returns {'a' => 2, 'b' => 2}.)

Now, if you require a Hash of Hashes, rather than an Array of Hashes, you can achieve this via the reduce function:

  $merged = $connections.reduce({}) |$memo, $x| {
    $memo + {$x[0] => merge($jdbc_default, $connections[$x[0]])}
  }

or:

  $merged = $connections.reduce({}) |$memo, $x| {
    $memo + {$x[0] => $jdbc_default + $connections[$x[0]]}
  }

How this works:

reduce iterates over each [key, value] pair from the Hash. The start value is the empty Hash {} that is passed as an argument to reduce.

In the first round, $memo is set to {}, and $x is set to the first [key, value] pair. The key is, therefore, given by $x[0].

In the subsequent rounds, $memo retains the value returned by the expression in the Lambda in the previous iteration, i.e. $memo + {$x[0] => $connections[$x[0]] + $jdbc_default}.

Showing this works:

Notice: Scope(Class[Profile::Jdbc]): {connection_name1 => {username => user1, password => pass1, testWhileIdle => true, testOnBorrow => true, testOnReturn => false, timeBetweenEvictionRunsMillis => 30000, maxActive => 20, maxWait => 10000, initialSize => 5, removeAbandonedTimeout => 600, removeAbandoned => false, logAbandoned => true, minEvictableIdleTimeMillis => 30001}, connection_name2 => {username => user2, password => pass2, testWhileIdle => true, testOnBorrow => true, testOnReturn => false, timeBetweenEvictionRunsMillis => 30000, maxActive => 20, maxWait => 10000, initialSize => 5, removeAbandonedTimeout => 600, removeAbandoned => false, logAbandoned => true, minEvictableIdleTimeMillis => 30001}}
Notice: Compiled catalog for alexs-macbook-pro.local in environment production in 0.12 seconds
Notice: Applied catalog in 0.02 seconds

Thanks to Henrik Lindberg for explaining this use of reduce!

See also the explanation given in the Ruby docs here.

On a related note, Henrik mentioned that Puppet 5 will contain a new function, tree_each,

that can iterate over a structure consisting of Array, Hash and Object containers. It can iterate in depth or breadth first order and there are options for controling what to include (containers and/or values and/or include the root of the tree). Other operations can be performed by chaining to other iterative functions for filter and map operations.

The pull request to add this feature is here.