CODEWITHSUNDEEP
dockerapache-spark
Venkat Ankam
Venkat Ankam

Reputation: 926

Spark Kubernetees building docker image fails

I'm trying out Spark on Kubernetes. Just downloaded Spark 2.4.3 on an EC2 instance in my VPC. I have setup my proxy in /etc/sysconfig/docker and able to import and run docker images from docker hub.

Command: bin/docker-image-tool.sh -t k8s-spark-2.4.3 build 

Sending build context to Docker daemon  261.4MB
Step 1/15 : FROM openjdk:8-alpine
 ---> a3562aa0b991
Step 2/15 : ARG spark_jars=jars
 ---> Using cache
 ---> 2f9744e85911
Step 3/15 : ARG img_path=kubernetes/dockerfiles
 ---> Using cache
 ---> ba02760dc2df
Step 4/15 : ARG k8s_tests=kubernetes/tests
 ---> Using cache
 ---> a630900ca584
Step 5/15 : RUN set -ex &&     apk upgrade --no-cache &&     apk add --no-cache bash tini libc6-compat linux-pam nss &&     mkdir -p /opt/spark &&     mkdir -p /opt/spark/work-dir &&     touch /opt/spark/RELEASE &&     rm /bin/sh &&     ln -sv /bin/bash /bin/sh &&     echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su &&     chgrp root /etc/passwd && chmod ug+rw /etc/passwd
 ---> Running in 6b2fe7f7fcc6
+ apk upgrade --no-cache
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz: network error (check Internet connection and firewall)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz: network error (check Internet connection and firewall)
+ apk add --no-cache bash tini libc6-compat linux-pam nss
OK: 103 MiB in 54 packages
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz: network error (check Internet connec                                                                tion and firewall)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz: network error (check Internet connection and firewall)
  bash (missing):
    required by: world[bash]
  libc6-compat (missing):
    required by: world[libc6-compat]
  linux-pam (missing):
    required by: world[linux-pam]
  tini (missing):
    required by: world[tini]
ERROR: unsatisfiable constraints:
The command '/bin/sh -c set -ex &&     apk upgrade --no-cache &&     apk add --no-cache bash tini libc6-compat linux-pam nss &&     mkdir -p /opt/spark &&     mkdir -p /opt/spark/work-dir &&     touch /opt/spark/RELEASE &&     rm /bin/sh &&     ln -sv /bin/bash /bin/sh &&     echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su &&     chgrp root /etc/passwd && chmod ug+rw /etc/passwd' returned a non-zero code: 4
Sending build context to Docker daemon  261.4MB
Step 1/9 : ARG base_img
Step 2/9 : FROM $base_img
pull access denied for spark, repository does not exist or may require 'docker login'
Sending build context to Docker daemon  261.4MB
Step 1/9 : ARG base_img
Step 2/9 : FROM $base_img
pull access denied for spark, repository does not exist or may require 'docker login'

Any idea how to fix this? I tried running this after 'docker login' as well. Result is same.

Upvotes: 1

Views: 731

Answers (2)

mried
mried

Reputation: 16

I had this exact same problem from within an Ubuntu 16.04 VM running on my laptop.

I was able to download the packages with curl and to visit the alpine repository directories using a web browser, so I know the issue was not the network connection.

Believe it or not, the solution was to restart the docker service running within the VM.

I found this as a suggested solution when searching for one of the apline package loading errors and it worked!

Upvotes: 0

bern.ar
bern.ar

Reputation: 56

Seems like your EC2 instance has no conection with internet

WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz:
network error (check Internet connection and firewall)

Upvotes: 1

Related Questions