CODEWITHSUNDEEP
springspring-cloud-dataflow
Raj_Twister
Raj_Twister

Reputation: 53

Limit the total number of tasks running at same time in spring cloud data flow?

Every time we launch a spring cloud task, it starts a new jvm (java.exe), so if 25 tasks are launched , then it will start 25 jvm.

I was wondering to see How to limit the total number of all tasks (running for all deployed jars) at the same time ?

Let’s say if I have to limit the total number of all tasks running at a time to 25. Is there any setting in SCDF we can do this ?

Please let me know

Upvotes: 4

Views: 1863

Answers (2)

pvrforpranavvr
pvrforpranavvr

Reputation: 2938

By default maximumTaskExecutions is 20. You can check the runningExecutionCount and maximumTaskExecutionsvalue using below url

http://localhost:9393/tasks/executions/current

Local Machine

If you are running spring-cloud dataflow server in local machine using jar file then you need to pass an argument like below.

java -jar spring-cloud-dataflow-server-<VERSION>.jar --spring.cloud.dataflow.task.platform.<PLATFORM-TYPE>.accounts.<ACCOUNT-NAME>.maximum-concurrent-tasks=<TASK-COUNT>

eg:

java -jar spring-cloud-dataflow-server-2.7.1.jar --spring.cloud.dataflow.task.platform.local.accounts.default.maximum-concurrent-tasks=1

KUBERNETES

If you are running spring cloud server in kubernetes platform then you need to change the configuration of Kubernetes task platforms

  • Go to src/kubernetes/server/server-config.yaml , add the property in exact place

    spring:
  cloud:
    dataflow:
      task:
        platform:
          <PLATFORM-TYPE>:
            accounts:
              <ACCOUNT-NAME>:
                 maximum-concurrent-tasks: <TASK-COUNT>

eg:

spring:
  cloud:
    dataflow:
      task:
        platform:
          kubernetes:
            accounts:
              dev:
                maximum-concurrent-tasks: 10
              qa:
                maximum-concurrent-tasks: 30
  • Then apply the config kubectl apply -f src/kubernetes/server/server-config.yaml
  • Then restart spring-cloud-dataflow server.

Note:

  • The PLATFORM-TYPE refers to one of the currently supported deployers: local, cloudfoundry, or kubernetes
  • ACCOUNT-NAME is the name of a configured platform account (default if no accounts are explicitly configured).

Reference

Upvotes: 0

Siddhant Sorann
Siddhant Sorann

Reputation: 323

In the newer version of SCDF there is a deployer property that limits the number of concurrent task executions. 

deployer.<appName>.kubernetes.maximumConcurrentTasks

By default, the value is 20.

Upvotes: 1

Related Questions