Docker swarm mode: scale down a node and remove services

Question

I have set of tasks for given service t1, t2, ..., tk across nodes N1, N2, ...Nw.

Due to lower usage, I do not need as many tasks as k. I need only l tasks (l < k).

In fact, I do not need w nodes so I want to start removing machines and pay less. Removing one machine at a time is fine.

Each service has its own state. The services are started in replicated mode.

1) How can I remove a single node and force the docker swarm not to recreate the same number of tasks for the service?

Notes:

• I can ensure that no work is rerouted to tasks running on a specific node, so removing the specific node is safe.
• This is the easiest solution, I will end up with w - 1 nodes and l services assuming that on the removed node was served k - l services.

or

2) How can I remove specific containers (tasks) from docker swarm and keep the number of replicas of the service lower by the number of removed tasks?

Notes:

• I assume that I already removed a node. The services from the node were redeployed to other nodes.
• I monitor myself the containers (tasks) which serve no traffic -> no state is needed to maintain

or

3) Any other solution?

Answer 1

To use a concrete example let's say you have 3 nodes and 9 tasks. You now want to go to 2 nodes and 6 tasks, without any unnecessary rescheduling (e.g. 2 modes and 9 tasks, or 3 modes and 6 tasks).

To scale down a service and 'drain' a node at the same time, you can do this:

docker service update --replicas 6 --constraint-add "node.hostname != node_to_be_removed_hostname" service_name

If your existing setup is balanced, this should only cause the tasks running on the host to be removed to be killed.

After this, you can proceed to (docker node update) drain the node, remove it from the swarm, and remove the constraint that has just been added.

Answer 2

To answer your questions

Q1-> You can simply drain the node in the cluster to verify if the services on the services are started on other nodes. Once they do you can safely remove the node from the swarm cluster. docker node update --availability drain <>

Q2-> You must have specified replica count while starting the services, you can simply scale it to a lower count.

docker service scale <>=<>