CODEWITHSUNDEEP
dockerdockerfile
Suman Srivastava
Suman Srivastava

Reputation: 19

Dockerfile cannot run cp command to move file inside container

Hi I am trying to download a file inside container and moving this file in specific location inside container. 

RUN wget https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar 
RUN cp gcs-connector-latest-hadoop2.jar /opt/spark-2.2.1-bin-hadoop2.7/jars/
RUN cp /opt/spark-2.2.1-bin-hadoop2.7/conf/spark-defaults.conf.template /opt/spark-2.2.1-bin-hadoop2.7/conf/spark-defaults.conf
RUN echo "spark.hadoop.google.cloud.auth.service.account.enable true" > /opt/spark-2.2.1-bin-hadoop2.7/conf/spark-defaults.conf

But this is failing with the error as below:

Step 44/46 : RUN cp gcs-connector-latest-hadoop2.jar /opt/spark-2.2.1-bin-hadoop2.7/jars/
 ---> Running in 8c81d9871377
cp: cannot create regular file '/opt/spark-2.2.1-bin-hadoop2.7/jars/': No such file or directory
The command '/bin/sh -c cp gcs-connector-latest-hadoop2.jar /opt/spark-2.2.1-bin-hadoop2.7/jars/' returned a non-zero code: 1

EDIT-1 Error Screenshot

I have tried the solution mentioned and now I am getting the below error:

Removing intermediate container e885431017e8 Step 43/44 : COPY /opt/spark-2.2.1-bin-hadoop2.7/conf/spark-defaults.conf.template /opt/spark-2.2.1-bin-hadoop2.7/conf/spark-defaults.conf lstat opt/spark-2.2.1-bin-hadoop2.7/conf/spark-defaults.conf.template: no such file or directory

Upvotes: 0

Views: 818

Answers (2)

LinPy
LinPy

Reputation: 18608

do you have already the path /opt/spark-2.2.1-bin-hadoop2.7/jars/ in your container?

if not add this before cp command:

mkdir -p /opt/spark-2.2.1-bin-hadoop2.7/jars/

then try to copy like this:

cp gcs-connector-latest-hadoop2.jar /opt/spark-2.2.1-bin-hadoop2.7/jars/gcs-connector-latest-hadoop2.jar

after your edit:

you run mkdir and try to copy from it , that should not work since the folder is empty !!

Upvotes: 2

Ntwobike
Ntwobike

Reputation: 2751

Why dont you download to the folder directly? and use COPY to copy files inside the container COPY

RUN wget https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar -P /opt/spark-2.2.1-bin-hadoop2.7/jars/
COPY /opt/spark-2.2.1-bin-hadoop2.7/conf/spark-defaults.conf.template /opt/spark-2.2.1-bin-hadoop2.7/conf/spark-defaults.conf

assumed: /opt/spark-2.2.1-bin-hadoop2.7/jars/folder exists

Upvotes: 0

Related Questions