CODEWITHSUNDEEP
apache-sparkpysparkmesosmesosphere
Ramakanta Samal
Ramakanta Samal

Reputation: 251

Running spark-submit in chronos

Currently I have a mesos cluster with docker container enabled. I use mesos docker support to run spark framework on my cluster. I want to schedule a spark-submit job from chronos. Could you please let me know the steps or the best way to do it. I am pretty new to mesos and I dont know how chronos will find the spark-submit command to submit the application.

Upvotes: 0

Views: 811

Answers (1)

Tobi
Tobi

Reputation: 31489

You could package your application together with a Spark distribution into a Docker container, and create a Docker job in Chronos:

For example, send a POST request to the Chronos REST API like this:

curl -L -H 'Content-Type: application/json' -X POST chronos-node:8080/scheduler/iso8601 -d '
{
  "schedule": "R/2016-04-15T12:00:00Z/PT2M",
  "name": "spark-submit-job",
  "container": {
    "type": "DOCKER",
    "image": "my/sparksubmitapp",
    "network": "BRIDGE",
    "forcePullImage": true
  },
  "cpus": "0.5",
  "mem": "1024",
  "uris": [],
  "command": "/path/to/spark/bin/spark-submit --class com.my.app.Main myApp.jar"
}'

Upvotes: 2

Related Questions