CODEWITHSUNDEEP
kubernetestriggersrabbitmqcelerykeda
Diego L
Diego L

Reputation: 908

KEDA ScaledObject not scaling multiple pods when messages arrive in shared RabbitMQ queue

I have the following situation: I have a Kubernetes cluster in which I have a Celery application that communicates using RabbitMQ. I have a pod that contains a Celery task but there are 3 different queues for this task:

I need two instances of this pod but each of these with different execution commands, these are the deploys:

First deploy:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker-forecast-model-training-deploy
  labels:
    role: worker-forecast-model-training-service
spec:
  replicas: 0
  selector:
    matchLabels:
      role: worker-forecast-model-training-service
      tier: web-service
  template:
    metadata:
      labels:
        role: worker-forecast-model-training-service
        tier: web-service
    spec:
      containers:
        - name: worker-forecast-model-training
          image: prueba-celery-keda
          imagePullPolicy: IfNotPresent
          command:
            - "celery"
          args: [
            "-A",
            "app.worker",
            "worker",
            "--without-gossip",
            "--without-mingle",
            "--without-heartbeat",
            "-l",
            "info",
            "--pool",
            "solo",
            "-Q",
            "training-forecast-dev,training-forecast-solo-dev"
          ]
          env:
            - name: C_FORCE_ROOT
              value: "True"
          resources:
            requests:
              memory: "80Mi"
              cpu: "80m"
            limits:
              memory: "11000Mi"
              cpu: "2"

Second deploy:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker-forecast-model-training-prefork-deploy
  labels:
    role: worker-forecast-model-training-prefork-service
spec:
  replicas: 0
  selector:
    matchLabels:
      role: worker-forecast-model-training-prefork-service
      tier: web-service
  template:
    metadata:
      labels:
        role: worker-forecast-model-training-prefork-service
        tier: web-service
    spec:
      containers:
        - name: worker-forecast-model-training-prefork
          image: prueba-celery-keda-prefork
          imagePullPolicy: IfNotPresent
          command:
            - "celery"
          args: [
            "-A",
            "app.worker",
            "worker",
            "--without-gossip",
            "--without-mingle",
            "--without-heartbeat",
            "-l",
            "info",
            "--pool",
            "prefork",
            "-Q",
            "training-forecast-dev,training-forecast-prefork-dev"
          ]
          env:
            - name: C_FORCE_ROOT
              value: "True"
          resources:
            requests:
              memory: "80Mi"
              cpu: "80m"
            limits:
              memory: "11000Mi"
              cpu: "2"

The only thing that changes is the launch command (the --pool which is an irrelevant Celery thing for this example and the queues it is listening to). As you can see, both share the training-forecast-dev queue, and in addition to this, each one has a different one.

So, I also have two ScaledObject:

First KEDA:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-forecast-model-deploy
spec:
  scaleTargetRef:
    name: worker-forecast-model-training-deploy
  pollingInterval: 10
  cooldownPeriod: 28800
  idleReplicaCount: 0
  minReplicaCount: 1
  maxReplicaCount: 1
  advanced:
    restoreToOriginalReplicaCount: true
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 60
          policies:
            - type: Percent
              value: 20
              periodSeconds: 1800
  triggers:
    - type: rabbitmq
      metadata:
        host: amqp://default_user:[email protected]:5672//
        queueName: training-forecast-dev
        mode: QueueLength
        value: "1"
    - type: rabbitmq
      metadata:
        host: amqp://default_user:[email protected]:5672//
        queueName: training-forecast-solo-dev
        mode: QueueLength
        value: "1"

Second KEDA:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-forecast-model-training-prefork-deploy
spec:
  scaleTargetRef:
    name: worker-forecast-model-training-prefork-deploy
  pollingInterval: 10
  cooldownPeriod: 28800
  idleReplicaCount: 0
  minReplicaCount: 1
  maxReplicaCount: 1
  advanced:
    restoreToOriginalReplicaCount: true
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 60
          policies:
            - type: Percent
              value: 20
              periodSeconds: 1800
  triggers:
    - type: rabbitmq
      metadata:
        host: amqp://default_user:[email protected]:5672//
        queueName: training-forecast-dev
        mode: QueueLength
        value: "1"
    - type: rabbitmq
      metadata:
        host: amqp://default_user:[email protected]:5672//
        queueName: training-forecast-prefork-dev
        mode: QueueLength
        value: "1"

Expected behavior I need that if a task arrives at training-forecast-prefork-dev or training-forecast-prefork-dev the corresponding pod is raised (this is running correctly). I also need that when a task arrives at training-forecast-dev, both pods are raised.

Current behavior When a task arrives at training-forecast-dev, only one of the pods is raised. I thought that since there were two ScaledObjects, both would be activated and the final result would be the execution of the two pods, but only one ends up being raised even if more than one task arrives at training-forecast-dev.

Upvotes: 0

Views: 370

Answers (1)

Rico
Rico

Reputation: 61621

You are using rabbitmq queues so it makes sense that only one pod is getting created because the message is getting consumed and deleted from the queue after being consumed. I would try using different queue names training-forecast-dev and maybe training-forecast-dev2.

If you want to have the same messages go into training-forecast-dev and training-forecast-dev2 you could take a look at a 'producing' a fan out example like this.

Upvotes: 0

Related Questions