Chef: delayed service notification

Question

This is my mongodb cookbook recipe:

node.default['mongodb3']['version'] = '3.4.2'
node.default['mongodb3']['repo'] = 'https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.4/x86_64/'

node.default['mongodb3']['config']['mongod']['net']['port'] = 30158
node.default['mongodb3']['config']['mongod']['net']['bindIp'] = 'localhost'
node.default['mongodb3']['config']['mongod']['security']['authorization'] = 'enabled'

include_recipe 'mongodb3::default'

cookbook_file "/tmp/setupUsers.js" do
  source "mongo/setupUsers.js"
  mode 0755
end

execute "Add Mongo Users" do
  command "mongo localhost:30158 /tmp/setupUsers.js"
end

As you can see:

1. I'm installing mongo 3.4 and,
2. I'm trying to perform mongo localhost:30158 /tmp/setupUsers.js.

However, I'm getting this message from chef:

==> default:   * execute[Add Mongo Users] action run
==> default:
==> default:     [execute] MongoDB shell version v3.4.2
==> default:               connecting to: localhost:30158
==> default:               2017-03-06T10:56:56.875+0000 W NETWORK  [thread1] Failed to connect to 127.0.0.1:30158, in(checking socket
for error after poll), reason: Connection refused
==> default:               2017-03-06T10:56:56.879+0000 E QUERY    [thread1] Error: couldn't connect to server localhost:30158, connec
tion attempt failed :
==> default:               connect@src/mongo/shell/mongo.js:237:13
==> default:               @(connect):1:6
==> default:               exception: connect failed
==> default:
==> default:     ================================================================================
==> default:     Error executing action `run` on resource 'execute[Add Mongo Users]'
==> default:     ================================================================================
==> default:
==> default:     Mixlib::ShellOut::ShellCommandFailed
==> default:     ------------------------------------
==> default:     Expected process to exit with [0], but received '1'
==> default:     ---- Begin output of mongo localhost:30158 /tmp/setupUsers.js ----
==> default:     STDOUT: MongoDB shell version v3.4.2
==> default:     connecting to: localhost:30158
==> default:     2017-03-06T10:56:56.875+0000 W NETWORK  [thread1] Failed to connect to 127.0.0.1:30158, in(checking socket for error
after poll), reason: Connection refused
==> default:     2017-03-06T10:56:56.879+0000 E QUERY    [thread1] Error: couldn't connect to server localhost:30158, connection attem
pt failed :
==> default:     connect@src/mongo/shell/mongo.js:237:13
==> default:     @(connect):1:6
==> default:     STDERR: exception: connect failed
==> default:     ---- End output of mongo localhost:30158 /tmp/setupUsers.js ----
==> default:     Ran mongo localhost:30158 /tmp/setupUsers.js returned 1
==> default:
==> default:     Resource Declaration:
==> default:     ---------------------
==> default:     # In /var/chef/cache/cookbooks/berk/recipes/security.rb
==> default:
==> default:       6: execute "Add Mongo Users" do
==> default:       7:   command "mongo localhost:30158 /tmp/setupUsers.js"
==> default:       8: end
==> default:
==> default:     Compiled Resource:
==> default:     ------------------
==> default:     # Declared in /var/chef/cache/cookbooks/berk/recipes/security.rb:6:in `from_file'
==> default:
==> default:     execute("Add Mongo Users") do
==> default:       action [:run]
==> default:       retries 0
==> default:       retry_delay 2
==> default:       default_guard_interpreter :execute
==> default:       command "mongo localhost:30158 /tmp/setupUsers.js"
==> default:       backup 5
==> default:       returns 0
==> default:       user nil
==> default:       declared_type :execute
==> default:       cookbook_name "berk"
==> default:       recipe_name "security"
==> default:     end
==> default:
==> default:     Platform:
==> default:     ---------
==> default:     x86_64-linux
==> default:

As you can see, It seems mongod service is not running yet, nevertheless at the end of the output message, chef is telling me that is trying to restart a delayed service notification.

==> default: [2017-03-06T10:56:56+00:00] INFO: Running queued delayed notifications before re-raising exception
==> default: [2017-03-06T10:56:56+00:00] INFO: template[/etc/mongod.conf] sending restart action to service[mongod] (delayed)
==> default: Recipe: mongodb3::default
==> default:   * service[mongod] action restart
==> default: [2017-03-06T10:56:57+00:00] INFO: service[mongod] restarted
==> default:
==> default:     - restart service service[mongod]
==> default: [2017-03-06T10:56:57+00:00] INFO: template[/opt/wildfly/standalone/configuration/standalone-full.xml] sending restart act
ion to service[wildfly] (delayed)

Why this service is not started until the end of chef configuration? I need to connect to mongo after mongod services is started

I've took a look on mongodb3 cookbook. According to this line (default recipe of mongodb3 recipe), the service should start immediatly.

EDIT

I'm using mongodb3 recipe. mongodb3 default recipe:

service 'mongod' do
  case node['platform']
    when 'ubuntu'
      if node['platform_version'].to_f >= 15.04
        provider Chef::Provider::Service::Systemd
      elsif node['platform_version'].to_f >= 14.04
        provider Chef::Provider::Service::Upstart
      end
  end
  supports :start => true, :stop => true, :restart => true, :status => true
  action :enable
  subscribes :restart, "template[#{node['mongodb3']['mongod']['config_file']}]", :delayed
  subscribes :restart, "template[#{node['mongodb3']['config']['mongod']['security']['keyFile']}", :delayed
end

Answer 1

I don't see service[mongod] anywhere in your recipe, but I suppose you have the following lines somewhere:

 service 'mongod' do
   action [:enable, :start]
 end

At this point Chef starts the service, but it may need some time to become fully running and responding to requests. Chef does not wait for that and continues to run the recipes.

You can try your execute[Add Mongo Users] several times, until it succeeds like this:

execute "Add Mongo Users" do
  command "mongo localhost:30158 /tmp/setupUsers.js"
  retries 6 #times
  retry_delay 10 #seconds
end

This will give your service a minute to start, before Chef will fail.

Important: You should also come up with some guard to the execute[Add Mongo Users], because otherwise it will run on every Chef run.

EDIT (After you showed the service[mongod] resource.)

It does not have the start action, that's why it is not started. So add the

service 'mongod' do
  action :start
end

somewhere before the execute[Add Mongo Users] resource.