Puppet: "Could not find class" hiccups *often* once after manifest/module changes

Question

I’ve got no idea if this is a bug or some kind of misuse/misconfiguration. Hope somebody can help. Thanks!

If I change a file in a module or manifest directory this often results in errors, mostly of the following kinds

• Could not find class for on node

• Cannot find definition Class on node

These errors occur when puppet modules/manifests are changed – but it happens only once. If it happens affected puppet agents will use their cached catalog once. The next time these agents request a catalog it is compiled flawlessly and everything will work like a charm – all errors are gone. Problem is we have agents with large catalogs. Their catalog runs can take up to over a minute and so the time it takes for a change to be applied will then be doubled to take up to 2 or even 3 minutes.

Usually changes of modules/manifests are performed by “git pull”. But it’s easy to reproduce these error messages by simple “touch” operations. Here are some interesting log observations and their [commonness]. Yes AFAICS they always appear in triplets. It rarely happens that there are no such problems at all after a modification.

> touch /opt/xxx/dev/puppet/manifests/site.pp

[often] puppet-master[12498]: Could not find class <class> for <node> on <node>
[often] puppet-master[12498]: Could not find class <class> for <node> on <node>
[often] puppet-master[12498]: Could not find class <class> for <node> on <node>

> touch /opt/xxx/dev/puppet/puppet.conf

[often] puppet-master[12498]: Could not find class <class> for <node> on <node>
[often] puppet-master[12498]: Could not find class <class> for <node> on <node>
[often] puppet-master[12498]: Could not find class <class> for <node> on <node>

[sometimes] puppet-master[12498]: Cannot find definition Class on node <node>
[sometimes] puppet-master[12498]: Cannot find definition Class on node <node>
[sometimes] puppet-master[12498]: Cannot find definition Class on node <node>

[rare] puppet-master[12498]: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type <own custom type> at <class file> on node <node>
[rare] puppet-master[12498]: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type <own custom type> at <class file> on node <node>
[rare] puppet-master[12498]: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type <own custom type> at <class file> on node <node>

> touch /opt/xxx/dev/puppet/modules/dispatcher/manifests/init.pp

[often] puppet-master[12498]: Could not find class <class not even part of module dispatcher> for <node not even affected by module dispatcher> on <node not even...>
[often] puppet-master[12498]: Could not find class <class not even part of module dispatcher> for <node not even affected by module dispatcher> on <node not even...>
[often] puppet-master[12498]: Could not find class <class not even part of module dispatcher> for <node not even affected by module dispatcher> on <node not even...>

Affected classes and nodes seem to be randomly chosen.

Environment this runs in:

> ll /etc/puppet
lrwxrwxrwx 1 root root   34 Mar  7 10:01 auth.conf -> /opt/xxx/dev/puppet/auth.conf
lrwxrwxrwx 1 root root   38 Mar  7 10:01 autosign.conf -> /opt/xxx/dev/puppet/autosign.conf
lrwxrwxrwx 1 root root   40 Mar  7 10:01 fileserver.conf -> /opt/xxx/dev/puppet/fileserver.conf
lrwxrwxrwx 1 root root   36 Mar  7 10:01 puppet.conf -> /opt/xxx/dev/puppet/puppet.conf

> ls -A /opt/xxx/dev/puppet/
auth.conf  autosign.conf  fileserver.conf  .git  .gitignore  manifests  modules  .project  puppet.conf  scripts

> cat /etc/puppet/puppet.conf # on puppetmaster
[main]
 logdir = /var/log/puppet
 rundir = /var/run/puppet
 ssldir = $vardir/ssl
 modulepath = /opt/xxx/dev/puppet/modules
 manifestdir = /opt/xxx/dev/puppet/manifests
 manifest = /opt/xxx/dev/puppet/manifests/site.pp

[agent]
 classfile = $vardir/classes.txt
 localconfig = $vardir/localconfig
 server = <puppetmaster host>
 report = true
 splaylimit = 0
 runinterval = 30

[master]
 certname=<puppetmaster host>
 reports = http,log
 reportdir = /var/lib/puppet/reports/upload
 reporturl = http://<puppetmaster host>:3000/reports

> cat /opt/xxx/dev/puppet/manifests/site.pp 
  import 'nodes.pp'
  $puppetserver = <puppetmaster host>

> rpm -qa | egrep "puppet|ruby"
  rubygem-rake-0.8.7-2.1.el6.noarch
  ruby-mysql-2.8.2-1.el6.x86_64
  libselinux-ruby-2.0.94-5.3.el6.x86_64
  puppet-3.1.0-1.el6.noarch
  rubygem-fastthread-1.0.7-2.el6.x86_64
  rubygem-mongrel-1.1.5-3.el6.x86_64
  ruby-1.8.7.352-7.el6_2.x86_64
  ruby-irb-1.8.7.352-7.el6_2.x86_64
  ruby-augeas-0.4.1-1.el6.x86_64
  ruby-shadow-1.4.1-13.el6.x86_64
  puppetlabs-release-6-6.noarch
  rubygems-1.3.7-1.el6.noarch
  puppet-server-3.1.0-1.el6.noarch
  rubygem-gem_plugin-0.2.3-3.el6.noarch
  rubygem-daemons-1.0.10-2.el6.noarch
  puppet-dashboard-1.2.22-1.el6.noarch
  ruby-libs-1.8.7.352-7.el6_2.x86_64
  ruby-rdoc-1.8.7.352-7.el6_2.x86_64
  rubygem-json-1.4.6-1.el6.x86_64

Answer 1

In my case that was locale issues on puppet master (some of my manifests contains localized comments), so Puppet 3.8 silently fails to compile them.

Check output of locale command in console (should not complain about anything).

I fixed it by running sudo dpkg-reconfigure locales and inserting next lines to /etc/default/locale:

LC_ALL="ru_RU.UTF-8"
LANG="ru_RU.UTF-8"
LANGUAGE="ru_RU:ru:en"

Also I'm using apache2 with passenger to run Puppet master so I also was need to replace the next string in /etc/init.d/apache2 from:

ENV="env -i LANG=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

to:

ENV="env -i LANG=ru_RU.UTF-8 LC_ALL=ru_RU.UTF-8 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

It turns out that Ruby launched under C locale wxpects all files to be in ASCII encoding.

Answer 2

Switching to an Apache/Passenger/Rack solution (and dumping WEBrick) seems to solve this problem. Additional info: http://projects.puppetlabs.com/issues/19642

Answer 3

I've had this happen when I've renamed a Puppet module/class, which results in:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class foo for example on node myhost.example.com
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

I have Puppet served with Passenger on Apache + Foreman, so the solution I've found is to give httpd and foreman-proxy a restart:

sudo service httpd restart
sudo service foreman-proxy restart

Also, if you use a code versioning system (as you should be :)) like git or svn, be sure you actually pushed your updates to the puppet master. In other words, check that your module actually exists where Puppet is looking for it, e.g. /etc/puppet/modules/foo. I've tripped myself up on this a few times.