CODEWITHSUNDEEP
kedakeda-scaledjob
MichelDelpech
MichelDelpech

Reputation: 863

Keda RabbitMQ - Keda not spawning additional jobs when queue has few messages

I have a Keda Scaledjob configured to spawn 1 job per message having the state 'ready' in RabbitMQ.

It has a max replica count set to 70.

Observed:

Here's my Keda configuration :

---
# Reference - https://keda.sh/docs/2.0/concepts/scaling-jobs/
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: scaledjob-puppeteer
  labels:
    environment: development
    app: puppeteer-display
spec:
  jobTargetRef:
    parallelism: 1                            # [max number of desired pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    completions: 1                            # [desired number of successfully finished pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    activeDeadlineSeconds: 7200               # (2 hours) Specifies the duration in seconds relative to the startTime that the job may be active before the system tries to terminate it; value must be positive integer
    backoffLimit: 2                           # Specifies the number of retries before marking this job failed. Defaults to 6
    template:
      spec:
        volumes:
          ...
        containers:
          ...
  pollingInterval: 10
  successfulJobsHistoryLimit: 0
  failedJobsHistoryLimit: 0
  maxReplicaCount: 75
  triggers:
    - type: rabbitmq
      metadata:
        protocol: amqp
        queueName: tasks
        mode: QueueLength
        value: "1"
      authenticationRef:
        name: keda-trigger-auth-rabbitmq-conn
---

How to make Keda to create a job whenever the queue has >= 1 message ?

Edit: It seems like it waits for at least 1 hour before creating the new job.

Upvotes: 0

Views: 1250

Answers (1)

P Z
P Z

Reputation: 11

The problem seems to be the missing scalingStrategy setting. You can add following configuration:

  scalingStrategy:
    strategy: accurate

The accurate setting is used when you consume messages from your queue instead of locking the messages. This is often used in other message queues.

For reference you can look into https://keda.sh/docs/2.7/concepts/scaling-jobs/ You can find further information about the scaling strategies in the details section.

Upvotes: 1

Related Questions