Kubernetes continuously killing and recreating last pod

Question

The last(3rd) container is continuously being delete and recreated by kubernetes. It goes from Running to Terminating state. The Kubernetes UI shows status as : 'Terminated: ExitCode:${state.terminated.exitCode}'

My deployment YAML:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: openapi
spec:
  scaleTargetRef:
    kind: Deployment
    name: openapi
  minReplicas: 3
  maxReplicas: 10
targetCPUUtilizationPercentage: 75

---
kind: Service
apiVersion: v1
metadata:
  name: openapi
spec:
  selector:
    app: openapi
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080
    - name: https
      protocol: TCP
      port: 443
      targetPort: 8443
 type: LoadBalancer

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: openapi
spec:
  template:
    metadata:
      labels:
        app: openapi
    spec:
      containers:
      - name: openapi
        image: us.gcr.io/PROJECT_ID/openapi:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8080

Portion of Output of kubectl get events -n namespace:

Pod                       Normal    Created                 kubelet                     Created container
Pod                       Normal    Started                 kubelet                     Started container
Pod                       Normal    Killing                 kubelet                     Killing container with id docker://openapi:Need to kill Pod
ReplicaSet                Normal    SuccessfulCreate        replicaset-controller       (combined from similar events): Created pod: openapi-7db5f8d479-p7mcl
ReplicaSet                Normal    SuccessfulDelete        replicaset-controller       (combined from similar events): Deleted pod: openapi-7db5f8d479-pgmxf
HorizontalPodAutoscaler   Normal    SuccessfulRescale       horizontal-pod-autoscaler    New size: 2; reason: Current number of replicas above Spec.MaxReplicas
HorizontalPodAutoscaler   Normal    SuccessfulRescale       horizontal-pod-autoscaler    New size: 3; reason: Current number of replicas below Spec.MinReplicas
Deployment                Normal    ScalingReplicaSet       deployment-controller        Scaled up replica set openapi-7db5f8d479 to 3
Deployment                Normal    ScalingReplicaSet       deployment-controller        Scaled down replica set openapi-7db5f8d479 to 2

kubectl describe pod -n default openapi-7db5f8d479-2d2nm for a pod that spawned and was killed:

A different pod with a different unique id spawns each time after a pod gets killed by Kubernetes.

Name:                      openapi-7db5f8d479-2d2nm
Namespace:                 default
Node:                      gke-testproject-default-pool-28ce3836-t4hp/10.150.0.2
Start Time:                Thu, 23 Nov 2017 11:50:17 +0000
Labels:                    app=openapi
                           pod-template-hash=3861948035
Annotations:               kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"openapi-7db5f8d479","uid":"b7b3e48f-ceb2-11e7-afe7-42010a960003"...
                           kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container openapi
Status:                    Terminating (expires Thu, 23 Nov 2017 11:51:04 +0000)
Termination Grace Period:  30s
IP:
Created By:                ReplicaSet/openapi-7db5f8d479
Controlled By:             ReplicaSet/openapi-7db5f8d479
Containers:
  openapi:
    Container ID:   docker://93d2f1372a7ad004aaeb34b0bc9ee375b6ed48609f505b52495067dd0dcbb233
    Image:          us.gcr.io/testproject-175705/openapi:latest
    Image ID:       docker-pullable://us.gcr.io/testproject-175705/openapi@sha256:54b833548cbed32db36ba4808b33c87c15c4ecde673839c3922577f30b
    Port:           8080/TCP
    State:          Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Thu, 23 Nov 2017 11:50:18 +0000
      Finished:     Thu, 23 Nov 2017 11:50:35 +0000
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        100m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-61k6c (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  default-token-61k6c:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-61k6c
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.alpha.kubernetes.io/notReady:NoExecute for 300s
                 node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason                 Age   From                                                Message
  ----    ------                 ----  ----                                                -------
  Normal  Scheduled              21s   default-scheduler                                   Successfully assigned openapi-7db5f8d479-2d2nm to gke-testproject-default-pool-28ce3836-t4hp
  Normal  SuccessfulMountVolume  21s   kubelet, gke-testproject-default-pool-28ce3836-t4hp  MountVolume.SetUp succeeded for volume "default-token-61k6c"
  Normal  Pulling                21s   kubelet, gke-testproject-default-pool-28ce3836-t4hp  pulling image "us.gcr.io/testproject-175705/openapi:latest"
  Normal  Pulled                 20s   kubelet, gke-testproject-default-pool-28ce3836-t4hp  Successfully pulled image "us.gcr.io/testproject-175705/openapi:latest"
  Normal  Created                20s   kubelet, gke-testproject-default-pool-28ce3836-t4hp  Created container
  Normal  Started                20s   kubelet, gke-testproject-default-pool-28ce3836-t4hp  Started container
  Normal  Killing                3s    kubelet, gke-testproject-default-pool-28ce3836-t4hp  Killing container with id docker://openapi:Need to kill Pod

Answer 1

Check the pod logs using the commands below: kubectl get events -w -n namespace and kubectl describe pod -n namespace pod_name