CODEWITHSUNDEEP
apache-sparkapache-zeppelin
William Fox
William Fox

Reputation: 1

Zeppelin 0.11.1 with Spark 3.3 - Spark Interpreter Fails with NoSuchFileException

Receiving an InterpreterException when submitting spark job through Zeppelin:

org.apache.zeppelin.interpreter.InterpreterException: java.io.IOException: Fail to launch interpreter process:
ERROR ClientEndpoint: Exception from cluster was: java.nio.file.NoSuchFileException: /zzz/ppp/lll/Zeppelin/zeppelin-0.11.1-bin-all/interpreter/spark/spark-interpreter-0.11.1.jar

The file is in the location. I changed ownership and gave it read write access but still ended up with the same outcome. I checked with the Spark Master and in the worker it states ava.nio.file.NoSuchFileException: /zzz/ppp/lll/Zeppelin/zeppelin-0.11.1-bin-all/interpreter/spark/spark-interpreter-0.11.1.jar so the request is making to the Workers on the Spark cluster but then errors out.

Upvotes: 0

Views: 88

Answers (1)

Gyeongtae Park
Gyeongtae Park

Reputation: 1

It is recommended to execute in the following order.

  1. Install and extract Spark binary files locally
# Install
wget https://archive.apache.org/dist/spark/spark-3.5.2/spark-3.5.2-bin-hadoop3.tgz

# Extract
tar -xvf spark-3.5.2-bin-hadoop3.tgz
  1. Create a Docker Compose file (docker-compose-with-spark.yml)
services:
  zeppelin:
    hostname: zeppelin
    container_name: zeppelin
    image: docker.io/apache/zeppelin:0.11.2
    ports:
      - "8080:8080"
    environment:
      ZEPPELIN_PORT: 8080
      ZEPPELIN_MEM: -Xmx1024m -XX:MaxMetaspaceSize=512m
      SPARK_HOME: /opt/spark
    volumes:
      - ./spark-3.5.2-bin-hadoop3:/opt/spark

  spark-master:
    hostname: spark-master
    container_name: spark-master
    image: docker.io/bitnami/spark:3.5.2
    ports:
      - "18080:8080"
      - "7077:7077"
    environment:
      SPARK_MODE: master
      SPARK_RPC_AUTHENTICATION_ENABLED: no
      SPARK_RPC_ENCRYPTION_ENABLED: no
      SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED: no
      SPARK_SSL_ENABLED: no
      SPARK_USER: spark
      SPARK_MASTER_PORT: 7077
      SPARK_MASTER_WEBUI_PORT: 8080

  spark-worker:
    hostname: spark-worker
    container_name: spark-worker
    image: docker.io/bitnami/spark:3.5.2
    ports:
      - "18081:8081"
    environment:
      SPARK_MODE: worker
      SPARK_MASTER_URL: spark://spark-master:7077
      SPARK_WORKER_MEMORY: 2G
      SPARK_WORKER_CORES: 2
      SPARK_RPC_AUTHENTICATION_ENABLED: no
      SPARK_RPC_ENCRYPTION_ENABLED: no
      SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED: no
      SPARK_SSL_ENABLED: no
      SPARK_USER: spark
      SPARK_WORKER_WEBUI_PORT: 8081
    depends_on:
      - spark-master
  1. Run Docker Compose
docker compose -f docker-compose-with-spark.yml up
  1. Create and run a Zeppelin notebook
%spark.conf

SPARK_HOME /opt/spark
spark.master spark://spark-master:7077

%spark

val sdf = spark.createDataFrame(Seq((0, "park", 13, 70, "Korea"), (1, "xing", 14, 80, "China"), (2, "john", 15, 90, "USA"))).toDF("id", "name", "age", "score", "country")
sdf.printSchema
sdf.show()

Result

root
 |-- id: integer (nullable = false)
 |-- name: string (nullable = true)
 |-- age: integer (nullable = false)
 |-- score: integer (nullable = false)
 |-- country: string (nullable = true)
+---+----+---+-----+-------+
| id|name|age|score|country|
+---+----+---+-----+-------+
|  0|park| 13|   70|  Korea|
|  1|xing| 14|   80|  China|
|  2|john| 15|   90|    USA|
+---+----+---+-----+-------+

Upvotes: 0

Related Questions