How to change kubelet options in RKE like eviction-hard (nodefs.available,imagefs.available,etc) in Rancher?

Question

I read carefully the Kubernetes Documentation here about extending the default 15% of imagefs.available and the others parameters but it doesn't say how to set it, i have installed the RKE (Rancher Kubernetes Engine) with the following configs.

Kubernets Version:v1.17.5
OS: Ubuntu 18.04.4 LTS
Container Runtime: 19.3.9

Btw the documentation doesn't clearly explain how to extend the default eviction so i come here to know if someone can tell me how to extend it .

I also tried kubelet command line but i get this

I0903 09:59:13.143727   31472 plugins.go:100] No cloud provider specified.
W0903 09:59:13.143760   31472 server.go:560] standalone mode, no API client
W0903 09:59:13.148508   31472 container_manager_linux.go:912] CPUAccounting not enabled for pid: 31472
W0903 09:59:13.148527   31472 container_manager_linux.go:915] MemoryAccounting not enabled for pid: 31472
W0903 09:59:13.579594   31472 server.go:474] No api server defined - no events will be sent to API server.
I0903 09:59:13.579620   31472 server.go:647] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
F0903 09:59:13.581308   31472 server.go:274] failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [Filename                           Type            Size    Used    Priority /dev/md0                                partition      16758780       9846712  -2]

Additional Information Server(Cloud,Bare metal,etc..): Bare Metal Kubernet installed using: rancher

when i do it by config.yaml with the kuberconfig file provided by rancher it try to do eviction of my containers in docker and not in kubernets containers.

So my command is something like this kubelet --kubeconfig=/root/.kube/config --config=/root/.kube/config.yaml

E0903 12:29:25.934582   26731 remote_image.go:135] RemoveImage "sha256:8c8e0533fa439319ae0ada18d3a7380d33f14f313ea5dd750ab68a1fde65d909" from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "rancher/rke-tools:v0.1.56" (must force) - container 2e31fd3c915d is using its referenced image 8c8e0533fa43
E0903 12:29:25.934602   26731 kuberuntime_image.go:120] Remove image "sha256:8c8e0533fa439319ae0ada18d3a7380d33f14f313ea5dd750ab68a1fde65d909" failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "rancher/rke-tools:v0.1.56" (must force) - container 2e31fd3c915d is using its referenced image 8c8e0533fa43
I0903 12:29:25.934614   26731 image_gc_manager.go:376] [imageGCManager]: Removing image "sha256:2b5329557072f7cd9591670db3814bb54a72aa474ca301f00c8a29597cd32c94" to free 18234790 bytes
E0903 12:29:25.935726   26731 remote_image.go:135] RemoveImage "sha256:2b5329557072f7cd9591670db3814bb54a72aa474ca301f00c8a29597cd32c94" from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "willfarrell/autoheal:latest" (must force) - container 5fb28a06d30d is using its referenced image 2b5329557072
E0903 12:29:25.935746   26731 kuberuntime_image.go:120] Remove image "sha256:2b5329557072f7cd9591670db3814bb54a72aa474ca301f00c8a29597cd32c94" failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "willfarrell/autoheal:latest" (must force) - container 5fb28a06d30d is using its referenced image 2b5329557072
I0903 12:29:25.935758   26731 image_gc_manager.go:376] [imageGCManager]: Removing image "sha256:0a3db21bc2b49b5cddea7ead8b99387ab37a332b71f0b016e1983587bb7d24af" to free 421267928 bytes
E0903 12:29:25.937276   26731 remote_image.go:135] RemoveImage "sha256:0a3db21bc2b49b5cddea7ead8b99387ab37a332b71f0b016e1983587bb7d24af" from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "linuxserver/plex:latest" (must force) - container 8ca7a3b6202b is using its referenced image 0a3db21bc2b4
E0903 12:29:25.937295   26731 kuberuntime_image.go:120] Remove image "sha256:0a3db21bc2b49b5cddea7ead8b99387ab37a332b71f0b016e1983587bb7d24af" failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "linuxserver/plex:latest" (must force) - container 8ca7a3b6202b is using its referenced image 0a3db21bc2b4
I0903 12:29:25.937307   26731 image_gc_manager.go:376] [imageGCManager]: Removing image "sha256:7aab9b2c0cd6d39e03815bf3429be845257a016c3a3f5fd9e22b1ec5c3e9f329" to free 291526564 bytes
E0903 12:29:25.938673   26731 remote_image.go:135] RemoveImage "sha256:7aab9b2c0cd6d39e03815bf3429be845257a016c3a3f5fd9e22b1ec5c3e9f329" from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "rancher/rancher-agent:v2.4.3" (must force) - container 9eb5c8663b7d is using its referenced image 7aab9b2c0cd6
E0903 12:29:25.938693   26731 kuberuntime_image.go:120] Remove image "sha256:7aab9b2c0cd6d39e03815bf3429be845257a016c3a3f5fd9e22b1ec5c3e9f329" failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "rancher/rancher-agent:v2.4.3" (must force) - container 9eb5c8663b7d is using its referenced image 7aab9b2c0cd6
W0903 12:29:25.938733   26731 eviction_manager.go:430] eviction manager: unexpected error when attempting to reduce inodes pressure: wanted to free 9223372036854775807 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "portainer/portainer:latest" (must force) - container 6193548811df is using its referenced image 2869fc110bf7, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "rancher/coreos-etcd:v3.4.3-rancher1" (must force) - container fd5c341853c0 is using its referenced image a0b920cf970d, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "rancher/hyperkube:v1.17.5-rancher1" (must force) - container 640e1d162e1e is using its referenced image ff99d966b0ee, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "hawkinzzz/dngele:latest" (must force) - container c16369f9385b is using its referenced image f3eb38637bfa, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "pyouroboros/ouroboros:latest" (must force) - container c0e2abfb3104 is using its referenced image 6bde18e83f41, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "linuxserver/emby:beta" (must force) - container 3def488e8543 is using its referenced image 95d10324f024, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "physk/rclone-mergerfs:latest" (must force) - container 9affc85735a1 is using its referenced image 80e6bae8490b, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "rancher/rancher:latest" (must force) - container 549b67e38726 is using its referenced image ddb8c914dee4, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "rancher/rke-tools:v0.1.56" (must force) - container 2e31fd3c915d is using its referenced image 8c8e0533fa43, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "willfarrell/autoheal:latest" (must force) - container 5fb28a06d30d is using its referenced image 2b5329557072, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "linuxserver/plex:latest" (must force) - container 8ca7a3b6202b is using its referenced image 0a3db21bc2b4, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "rancher/rancher-agent:v2.4.3" (must force) - container 9eb5c8663b7d is using its referenced image 7aab9b2c0cd6]
I0903 12:29:25.951254   26731 eviction_manager.go:355] eviction manager: must evict pod(s) to reclaim inodes
I0903 12:29:25.951274   26731 eviction_manager.go:373] eviction manager: pods ranked for eviction: canal-l29wx_kube-system(421796dc-24b3-4bcb-97f3-f61e36665c05)
E0903 12:29:25.951289   26731 eviction_manager.go:575] eviction manager: cannot evict a critical pod canal-l29wx_kube-system(421796dc-24b3-4bcb-97f3-f61e36665c05)
I0903 12:29:25.951297   26731 eviction_manager.go:396] eviction manager: unable to evict any pods from the node

Kind regards

Answer 1

The kubelet has the following default hard eviction threshold: memory.available<100Mi nodefs.available<10% nodefs.inodesFree<5% imagefs.available<15%

As per official Rancher page:

You can add additional arguments/binds/environment variables via the Config File option in Cluster Options. For more information, see the Extra Args, Extra Binds, and Extra Environment Variables in the RKE documentation or browse the Example Cluster.ymls.

Look in the full example how you can configure kubelet options:

kubelet:
      # Base domain for the cluster
      cluster_domain: cluster.local
      # IP address for the DNS service endpoint
      cluster_dns_server: 10.43.0.10
      # Fail if swap is on
      fail_swap_on: false
      # Set max pods to 250 instead of default 110
      extra_args:
        max-pods: 250

Here you can add smth like below or whatever you exactly need.

eviction-hard: "nodefs.available<1%,imagefs.available<1%"
eviction-soft-grace-period: "nodefs.available=30s,imagefs.available=30s"
eviction-soft: "nodefs.available<1%,imagefs.available<1%"

Frankly speaking im not sure if you can change RKE's kubelet on flight, but for sure you can create cluster from scratch with required options.