Reputation: 509
We run Airflow on K8s on DigitalOcean using Helm Chart. Tasks are written using airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator
. On a regular basis we see that pods are failing with the following messages (but the issue is not constant, so the majority of time it works fine):
We are using KubernetesExecutor
.
Helm Chart info - airflow-stable/airflow version 7.16.0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 31s default-scheduler Successfully assigned airflow/... to k8s-sai-pool-8c13r-9b9-8br6p
Normal Pulling 18s (x2 over 31s) kubelet Pulling image "registry.digitalocean.com/...:455eebb0"
Warning Failed 18s (x2 over 30s) kubelet Failed to pull image "registry.digitalocean.com/...:455eebb0": rpc error: code = Unknown desc = Error response from daemon: Get https://{URL}: unauthorized: authentication required
Warning Failed 18s (x2 over 30s) kubelet Error: ErrImagePull
Normal SandboxChanged 18s (x7 over 30s) kubelet Pod sandbox changed, it will be killed and re-created.
Normal BackOff 16s (x6 over 29s) kubelet Back-off pulling image "registry.digitalocean.com/...:455eebb0"
Warning Failed 16s (x6 over 29s) kubelet Error: ImagePullBackOff
Upvotes: 1
Views: 315
Reputation: 21
That are most likely just DigitalOcean Registry issues I suggest to try dockerhub
Upvotes: 2