Cannot find spark image built with docker-image-tool locally

Question

I am trying to build spark image to use with Kubernetes (minikube locally) (following https://spark.apache.org/docs/latest/running-on-kubernetes.html, https://itnext.io/running-spark-job-on-kubernetes-minikube-958cadaddd55 and https://learn.microsoft.com/en-us/azure/aks/spark-job). I want to build it for local docker repo and I am using following command ./bin/docker-image-tool.sh -m -t v2 build to build it for minikube. This build succeeds, but after that I can't find the image with docker image ps. The spark-submit fails with cryptic Error: Could not find or load main class org.apache.spark.launcher.Main, which doesn't tell anything, but it seems reasonable to be linked with image missing in the docker registry.

All build steps seem to succeed with no issue, so I am bit lost at what went wrong.

Build output:

Sending build context to Docker daemon  1.193GB
Step 1/15 : FROM openjdk:8-jdk-slim
 ---> 6b2212023a6e
Step 2/15 : ARG spark_jars=jars
 ---> Using cache
 ---> 2407ea1f6efa
Step 3/15 : ARG img_path=kubernetes/dockerfiles
 ---> Using cache
 ---> db7404f15dfd
Step 4/15 : ARG k8s_tests=kubernetes/tests
 ---> Using cache
 ---> 8dc19efc9230
Step 5/15 : RUN set -ex &&     apt-get update &&     ln -s /lib /lib64 &&     apt install -y bash tini libc6 libpam-modules libnss3 &&     mkdir -p /opt/spark &&     mkdir -p /opt/spark/work-dir &&     touch /opt/spark/RELEASE &&     rm /bin/sh &&     ln -sv /bin/bash /bin/sh &&     echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su &&     chgrp root /etc/passwd && chmod ug+rw /etc/passwd &&     rm -rf /var/cache/apt/*
 ---> Using cache
 ---> 5884173cd551
Step 6/15 : COPY ${spark_jars} /opt/spark/jars
 ---> Using cache
 ---> ff058e0c80e7
Step 7/15 : COPY bin /opt/spark/bin
 ---> Using cache
 ---> d158732d730c
Step 8/15 : COPY sbin /opt/spark/sbin
 ---> Using cache
 ---> 896cd481cdbc
Step 9/15 : COPY ${img_path}/spark/entrypoint.sh /opt/
 ---> Using cache
 ---> 1d108a1e1df4
Step 10/15 : COPY examples /opt/spark/examples
 ---> Using cache
 ---> 3a7680f04b03
Step 11/15 : COPY ${k8s_tests} /opt/spark/tests
 ---> Using cache
 ---> 2cd76b303d3e
Step 12/15 : COPY data /opt/spark/data
 ---> Using cache
 ---> 8221e20aac95
Step 13/15 : ENV SPARK_HOME /opt/spark
 ---> Using cache
 ---> ecb9bee32447
Step 14/15 : WORKDIR /opt/spark/work-dir
 ---> Using cache
 ---> 6bfafec8d0bb
Step 15/15 : ENTRYPOINT [ "/opt/entrypoint.sh" ]
 ---> Using cache
 ---> 110ccecb98db
Successfully built 110ccecb98db
Successfully tagged spark:v2
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.
Sending build context to Docker daemon  1.193GB
Step 1/9 : ARG base_img
Step 2/9 : FROM $base_img
 ---> 110ccecb98db
Step 3/9 : WORKDIR /
 ---> Using cache
 ---> 7d82c43a71b8
Step 4/9 : RUN mkdir ${SPARK_HOME}/python
 ---> Using cache
 ---> a06db167de65
Step 5/9 : RUN apt install -y python python-pip &&     apt install -y python3 python3-pip &&     rm -r /usr/lib/python*/ensurepip &&     pip install --upgrade pip setuptools &&     rm -r /root/.cache && rm -rf /var/cache/apt/*
 ---> Using cache
 ---> 2789da56886a
Step 6/9 : COPY python/lib ${SPARK_HOME}/python/lib
 ---> Using cache
 ---> 17ad5567580c
Step 7/9 : ENV PYTHONPATH ${SPARK_HOME}/python/lib/pyspark.zip:${SPARK_HOME}/python/lib/py4j-*.zip
 ---> Using cache
 ---> 01f0ee91d6a6
Step 8/9 : WORKDIR /opt/spark/work-dir
 ---> Using cache
 ---> 95b4885abc7e
Step 9/9 : ENTRYPOINT [ "/opt/entrypoint.sh" ]
 ---> Using cache
 ---> 2bb9c23681e8
Successfully built 2bb9c23681e8
Successfully tagged spark-py:v2
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.
Sending build context to Docker daemon  1.193GB
Step 1/9 : ARG base_img
Step 2/9 : FROM $base_img
 ---> 110ccecb98db
Step 3/9 : WORKDIR /
 ---> Using cache
 ---> 7d82c43a71b8
Step 4/9 : RUN mkdir ${SPARK_HOME}/R
 ---> Using cache
 ---> c49d10da5c74
Step 5/9 : RUN apt install -y r-base r-base-dev && rm -rf /var/cache/apt/*
 ---> Using cache
 ---> d693ef1ab12c
Step 6/9 : COPY R ${SPARK_HOME}/R
 ---> Using cache
 ---> 50ca4b703501
Step 7/9 : ENV R_HOME /usr/lib/R
 ---> Using cache
 ---> cba4fbc51e7c
Step 8/9 : WORKDIR /opt/spark/work-dir
 ---> Using cache
 ---> 449edacaae16
Step 9/9 : ENTRYPOINT [ "/opt/entrypoint.sh" ]
 ---> Using cache
 ---> a84c327a2b9d
Successfully built a84c327a2b9d
Successfully tagged spark-r:v2
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.

Cannot find spark image built with docker-image-tool locally

Answers (1)

Related Questions