CODEWITHSUNDEEP
apache-sparkconfigurationspark-jobserver
Kiran
Kiran

Reputation: 223

Spark Job-Server configuaration in StandAlone cluster

I am trying to set up a Spark JobServer (SJS) to execute jobs on a Standalone Spark cluster. I am trying to deploy SJS on one of the non-master nodes of SPARK cluster. I am not using the docker, but trying to do manually.

I am confused with the help documents in SJS github particulary the deployment section. Do I need to edit both local.conf and local.sh to run this?

Can someone point out the steps to set up the SJS in the spark cluster?

Thanks! Kiran

Update: I created a new environment to deploy jobserver in one of the nodes of the cluster: Here are the details of it:

env1.sh:

DEPLOY_HOSTS="masked.mo.cpy.corp"
APP_USER=kiran
APP_GROUP=spark
INSTALL_DIR=/home/kiran/job-server
LOG_DIR=/var/log/job-server
PIDFILE=spark-jobserver.pid
JOBSERVER_MEMORY=1G
SPARK_VERSION=1.6.1
MAX_DIRECT_MEMORY=512M
SPARK_HOME=/home/spark/spark-1.6.1-bin-hadoop2.6
SPARK_CONF_DIR=$SPARK_HOME/conf
SCALA_VERSION=2.11.6

env1.conf

spark {

  master = "local[1]"
  webUrlPort = 8080
  job-number-cpus = 2

  jobserver {
    port = 8090
    bind-address = "0.0.0.0"
    jar-store-rootdir = /tmp/jobserver/jars
    context-per-jvm = false
    jobdao = spark.jobserver.io.JobFileDAO
    filedao {
      rootdir = /tmp/spark-job-server/filedao/data
    }
    datadao {
      rootdir = /tmp/spark-jobserver/upload
    }

    result-chunk-size = 1m
  }

  context-settings {
    num-cpu-cores = 1
    memory-per-node = 1G
  }
  home = "/home/spark/spark-1.6.1-bin-hadoop2.6"
}

Upvotes: 0

Views: 1467

Answers (2)

noorul
noorul

Reputation: 1353

Why don't you set JOBSERVER_FG=1 and try running server_start.sh, this would run the process in foreground and should display the error to stderr.

Upvotes: 1

Daniel de Paula
Daniel de Paula

Reputation: 17862

Yes, you have edit both files adapting them for your cluster.

The deploy steps are explained below:

  1. Copy config/local.sh.template to <environment>.sh and edit as appropriate.

This file is mostly for environment variables that are used by the deployment script and by the server_start.sh script. The most important ones are: deploy host (it's the ip or hostname where the jobserver will be run), user and group of execution, JobServer memory (it will be the driver memory), spark version and spark home.

  1. Copy config/shiro.ini.template to shiro.ini and edit as appropriate. NOTE: only required when authentication = on

If you are going to use shiro authentication, then you need this step.

  1. Copy config/local.conf.template to <environment>.conf and edit as appropriate.

This is the main configuration file for JobServer and for the contexts that JobServer will create. The full list of the properties you can set in this file can be seen on this link.

  1. bin/server_deploy.sh <environment>

After editing the configuration files, you can deploy using this script. The parameter must be the name that you chose for your .conf and .sh files.

Once you run the script, JobServer will connect to the host entered in the .sh file and will create a new directory with some control files. Then, every time you need to change a configuration entry, you can do it directly on the remote machine: the .conf file will be there with the name you chose and the .sh file will be renamed to settings.sh.

Please note that, if you haven't configured an SSH key based connection between the machine where you run this script and the remote machine, you will be prompted for password during its execution.

If you have problems with the creation of directories on the remote machine, you can try and create them yourself with mkdir (they must match the INSTALL_DIR configuration entry of the .sh file) and change their owner user and group to match the ones entered in the .sh configuration file.

  1. On the remote server, start it in the deployed directory with server_start.sh and stop it with server_stop.sh

This is very informative. Once you have done all other steps, you can start JobServer service on the remote machine by running the script server_start.sh and you can stop it with server_stop.sh

Upvotes: 0

Related Questions