CODEWITHSUNDEEP
pythondockerdockerfile
Sergey Ivanov
Sergey Ivanov

Reputation: 3929

Run python script that imports local script in a docker

I have a local script main.py that imports another local script submain.py. When I run it in a docker I get the following error:

qsub -cwd -soft -l docker,docker_images="*docker_imagename*" -S /usr/bin/python ./main.py --arg1 value1 --arg2 value2

from submain import func
ImportError: No module named submain

My Dockerfile looks like this: 

FROM ubuntu:latest
RUN apt-get -y update && apt-get -y install build-essential libxml2-dev zlib1g-dev python-dev python-pip pkg-config libffi-dev libcairo-dev
RUN pip install --upgrade pip
RUN pip install python-igraph scikit-learn numpy scipy matplotlib

CMD /usr/local/bin/igraph

How can I run my script main.py with additional script stored locally? If this not possible, how I "attach" submain.py to docker image?

Upvotes: 0

Views: 1643

Answers (2)

Roberto Salgado
Roberto Salgado

Reputation: 71

In answer to

How can I run my script main.py with additional script stored locally?

Python will try to load a module using its libraries path and current dir. So, since you seem to be using /root/ as main folder, you could add this line to a docker run execution (I don't have experience with qsub):

-v /local/path/to/submain.py:/root/submain.py

It will map container's /root/submain.py to your local submain.py. So you don't need to copy your submain file to the docker image.

Mind that you are running your scripts as root inside a container. So any volume or data that the container can write/read could be exposed if you run any insecure service.

It's also a good idea to check WORKDIR and USER Dockerfile options.

In answer to

If this not possible, how I "attach" submain.py to docker image?

using this inside the Dockerfile will do the trick:

COPY submain.py submain.py

Mind that you must build a new image before every in-container testing or use a volume as in previous example before sending it to the grid.

Upvotes: 0

dshockley
dshockley

Reputation: 1494

Here's a minimal example of how this works, with docker + python (ignoring qsub).

First, main.py:

from submain import my_fn

if __name__ == '__main__':
    print('got val {} from submain.my_fn'.format(my_fn(12)))

And submain.py:

def my_fn(val):
    return val * 2

And the Dockerfile [1]:

FROM python:3

COPY main.py main.py
COPY submain.py submain.py

ENTRYPOINT python main.py

Then to test it:

$ docker build -t main-py-img .
# ... lots of output ...
Successfully built e79194e43094
Successfully tagged main-py-img:latest
$ docker run main-py-img

If possible, you should try to test this on a local environment, to work out the python + docker issues, before you move to submitting a job with qsub. Using qsub to submit a job that runs your docker image is a different issue -- probably you should create a new question about that, once you get this part resolved.

[1] For simplicity I've inherited from the python:3 docker image, to avoid having to install all the python dependencies. If for some reason you need Ubuntu, you can try with that instead. In case you don't need Ubuntu, you can just add RUN pip install igraph to the Dockerfile above (inherit from python:2 if you need python 2 instead of python 3), but I would recommend to start with the most minimal example possible, and build from there.

Upvotes: 1

Related Questions