Docker container run error: Container exited on error: exited with status 1

Question

I am running dependency application in marathon, one is mongo DB and other is karesti voting app. I have deployed these applications in marathon within a docker container. First I deployed mongo db and later the voting app.

mongo db docker image is working fine in marathon. the issue is with voting app.... voting app is switching between the running mode and suspend mode continuously. Due to this the port number is also getting changed in each deployment and a new container is creating the docker for the same image.

Here I have mesos DNS for service discovery.

Error from marathon:

Docker container run error: Container exited on error: exited with status 1

Error from voting app:

Received killTask for task cmvoting.dff585d5-2bc5-11e6-a4c0-0242efa3f3aa

Log message from mesos slave: 
I0607 07:24:04.297161  1957 slave.cpp:3243] Handling status update TASK_RUNNING (UUID: ed039d5b-ef25-45e5-962f-367feaa33ae5) for task voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3 of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001 from executor(1)@127.0.0.1:50983
I0607 07:24:04.299815  1957 status_update_manager.cpp:320] Received status update TASK_RUNNING (UUID: ed039d5b-ef25-45e5-962f-367feaa33ae5) for task voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3 of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001
I0607 07:24:04.300282  1957 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_RUNNING (UUID: ed039d5b-ef25-45e5-962f-367feaa33ae5) for task voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3 of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001
I0607 07:24:04.391424  1957 slave.cpp:3641] Forwarding the update TASK_RUNNING (UUID: ed039d5b-ef25-45e5-962f-367feaa33ae5) for task voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3 of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001 to master@172.31.29.198:5050
I0607 07:24:04.391826  1957 slave.cpp:3551] Sending acknowledgement for status update TASK_RUNNING (UUID: ed039d5b-ef25-45e5-962f-367feaa33ae5) for task voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3 of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001 to executor(1)@127.0.0.1:50983
I0607 07:24:04.421507  1951 status_update_manager.cpp:392] Received status update acknowledgement (UUID: ed039d5b-ef25-45e5-962f-367feaa33ae5) for task voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3 of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001
I0607 07:24:04.421766  1951 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_RUNNING (UUID: ed039d5b-ef25-45e5-962f-367feaa33ae5) for task voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3 of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001


I0607 07:24:18.299789  1956 slave.cpp:3243] Handling status update TASK_FAILED (UUID: e20d07aa-88e1-497b-9c51-79252c279f27) for task voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3 of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001 from executor(1)@127.0.0.1:50983
I0607 07:24:18.321115  1953 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID: e20d07aa-88e1-497b-9c51-79252c279f27) for task voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3 of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001
I0607 07:24:18.321229  1953 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_FAILED (UUID: e20d07aa-88e1-497b-9c51-79252c279f27) for task voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3 of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001
I0607 07:24:18.338953  1956 slave.cpp:3641] Forwarding the update TASK_FAILED (UUID: e20d07aa-88e1-497b-9c51-79252c279f27) for task voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3 of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001 to master@172.31.29.198:5050
I0607 07:24:18.339341  1956 slave.cpp:3551] Sending acknowledgement for status update TASK_FAILED (UUID: e20d07aa-88e1-497b-9c51-79252c279f27) for task voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3 of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001 to executor(1)@127.0.0.1:50983
I0607 07:24:18.350075  1954 status_update_manager.cpp:392] Received status update acknowledgement (UUID: e20d07aa-88e1-497b-9c51-79252c279f27) for task voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3 of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001
I0607 07:24:18.350272  1954 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_FAILED (UUID: e20d07aa-88e1-497b-9c51-79252c279f27) for task voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3 of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001
I0607 07:24:19.301136  1951 slave.cpp:3769] executor(1)@127.0.0.1:50983 exited
I0607 07:24:19.327797  1953 docker.cpp:2011] Executor for container 'd4447d21-23da-45db-a9aa-f6b2d75e2db7' has exited
I0607 07:24:19.327910  1953 docker.cpp:1731] Destroying container 'd4447d21-23da-45db-a9aa-f6b2d75e2db7'
I0607 07:24:19.328034  1953 docker.cpp:1859] Running docker stop on container 'd4447d21-23da-45db-a9aa-f6b2d75e2db7'
I0607 07:24:19.328732  1958 slave.cpp:4120] Executor 'voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3' of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001 exited with status 0
I0607 07:24:19.328843  1958 slave.cpp:4224] Cleaning up executor 'voting.cfc5c791-2c80-11e6-b454-0242f9dcd7c3' of framework 4925c40b-fb4b-4ec7-91fc-3ffc16e1bac8-0001 at executor(1)@127.0.0.1:50983

I am not able to identify where is the issue, why the voting app is been failed in slave.

If I run the voting from bare ubuntu machine, I am able to access the application, which has linked to the database and everything is working fine. Which shows that no issue with the app, no issue with the docker image.

My suspect is that, something is happening in mesos slave which is not able to handle or might not be able to connect to the database.

my mongo db listen to the the port 27017 and voting app listen to port 9000. I can see the only the port 27017 is being used when i do the netstat -anp | grep 27017

The command used to run the slave is

sudo ./bin/mesos-slave.sh --master=zk://172.31.36.222:2181/home/ubuntu/mesos --executor_registration_timeout=5mins --containerizers="docker,mesos"

The command used to run the marathon is

./bin/start --master zk://172.31.36.222:2181/home/ubuntu/mesos --zk zk://172.31.36.222:2181/home/ubuntu/marathon-1.1.1 --task_launch_timeout 600000

Please let me know if you need any more info or clarity on the problem

Thanks in advance for your kind help....

Answer 1

I thanks for the response I have got...

I found the solution to this problem

The DNS looks for the right IP address from the files /etc/resolv.conf and /etc/hosts

I removed the localhost IP address from /etc/hosts file, where DNS was trying to resolve from 127.0.0.1 which is unknown to external world. I provide my VM private IP address which same has been provided in the DNS conf file

Answer 2

One part of the problem could be that your slave is binding to the 127.0.0.1 ip address. Please try to set a non-local address on your slave like this:

echo 172.31.36.223 | sudo tee /etc/mesos-slave/ip
sudo cp /etc/mesos-slave/ip /etc/mesos-slave/hostname

Replace 172.31.36.223 with the real ip address of your slave.

Another part could probably be that the voting app can't connect to MongoDB because the port number is different. You should also post your Marathon app definitions to be able to verify this.

If you just have one slave, you could run the both apps in HOST networking mode, so that you can define the port numbers statically. If you're using bridged mode, you need to employ some kind of service discovery mechanism so that the voting app can find the MongoDB server.