CODEWITHSUNDEEP
pythondockerrustalpine-linuxorjson
vivekv
vivekv

Reputation: 2298

How do I setup rust toolchain for orjson python library build in an alpine docker image?

I have been struggling with this and I need some help. I am trying to create a docker image of my python app. My App uses orjson, the fast JSON library for python. Parts of this library is built using rust so I need the rust toolchain. Now the kicker is that I am using Alpine for its small footprint so it comes with none of the standard tools included. I need set things up myself.

Here is my Dockerfile to simulate the problem.

FROM python:3.7-alpine

# for orjson in requirements.txt
RUN apk add rust cargo


RUN pip install orjson

This is the error message that I am facing.

Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM python:3.7-alpine
 ---> 6a5ca85ed89b
Step 2/3 : RUN apk add rust cargo
 ---> Running in b86315a52e50
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/19) Installing rust-stdlib (1.43.1-r1)
(2/19) Installing libgcc (9.3.0-r2)
(3/19) Installing libstdc++ (9.3.0-r2)
(4/19) Installing binutils (2.34-r1)
(5/19) Installing gmp (6.2.0-r0)
(6/19) Installing isl (0.18-r0)
(7/19) Installing libgomp (9.3.0-r2)
(8/19) Installing libatomic (9.3.0-r2)
(9/19) Installing libgphobos (9.3.0-r2)
(10/19) Installing mpfr4 (4.0.2-r4)
(11/19) Installing mpc1 (1.1.0-r1)
(12/19) Installing gcc (9.3.0-r2)
(13/19) Installing musl-dev (1.1.24-r8)
(14/19) Installing libxml2 (2.9.10-r4)
(15/19) Installing llvm10-libs (10.0.0-r2)
(16/19) Installing rust (1.43.1-r1)
(17/19) Installing nghttp2-libs (1.41.0-r0)
(18/19) Installing libcurl (7.69.1-r0)
(19/19) Installing cargo (1.43.1-r1)
Executing busybox-1.31.1-r16.trigger
OK: 334 MiB in 54 packages
Removing intermediate container b86315a52e50
 ---> 07671da6f533
Step 3/3 : RUN pip install orjson
 ---> Running in 9c72ff2b2e3e
Collecting orjson
  Downloading orjson-3.0.2.tar.gz (649 kB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpxhkdxsol
         cwd: /tmp/pip-install-0x1gorur/orjson
    Complete output (14 lines):
    💥 maturin failed
      Caused by: Cargo metadata failed. Do you have cargo in your PATH?
      Caused by: Error during execution of `cargo metadata`: error: failed to run `rustc` to learn about target-specific information

    Caused by:
      process didn't exit successfully: `rustc - --crate-name ___ --print=file-names -Z mutable-noalias --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit code: 1)
    --- stderr
    error: the option `Z` is only accepted on the nightly compiler



    Checking for Rust toolchain....
    Running `maturin pep517 write-dist-info --metadata-directory /tmp/pip-modern-metadata-7txy00_d --manylinux=off --strip=on`
    Error: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/tmp/pip-modern-metadata-7txy00_d', '--manylinux=off', '--strip=on']' returned non-zero exit status 1.
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpxhkdxsol Check the logs for full command output.
The command '/bin/sh -c pip install orjson' returned a non-zero code: 1

I am a bit lost on what I should install for getting a rust environment in Alpine. Any help would be appreciated.

Upvotes: 1

Views: 9115

Answers (3)

Magus
Magus

Reputation: 2992

Most distros use glibc, alpine uses musl and that's the reason you have to recompile everything. If you are really in the need of building a production grade container, yeah, you're kind of doomed to find and install all the dependencies.

If you're like me and you're only getting your dev env running, just switch from alpine to stretch and all your problems will go away.

Here's the source:

https://github.com/ijl/orjson/issues/98#issuecomment-643248331

Upvotes: 1

Nitishkumar Singh
Nitishkumar Singh

Reputation: 1839

Please follow this github issue discussion to get more clarity on it.

Dockerfile

FROM alpine:3.10
RUN echo "https://dl-3.alpinelinux.org/alpine/v3.10/main" >> /etc/apk/repositories
RUN echo "https://dl-3.alpinelinux.org/alpine/v3.10/community" >> /etc/apk/repositories
RUN apk add --no-cache python3 gcompat patchelf
RUN patchelf --add-needed libgcompat.so.0 /usr/bin/python3.7
RUN echo 'manylinux1_compatible = True' > /usr/lib/python3.7/_manylinux.py &&\
    pip3 install orjson &&\
    rm /usr/lib/python3.7/_manylinux.py

Upvotes: 3

vivekv
vivekv

Reputation: 2298

I managed to get it to work thanks to @nitishkumar-singh response. There were some errors in that post here is my Dockerfile if someone runs into this issue.

FROM python:3.7-alpine3.11

RUN echo "https://dl-3.alpinelinux.org/alpine/v3.11/main" >> /etc/apk/repositories
RUN echo "https://dl-3.alpinelinux.org/alpine/v3.11/community" >> /etc/apk/repositories
RUN apk add --no-cache python3 gcompat patchelf
RUN patchelf --add-needed libgcompat.so.0 /usr/bin/python3
RUN echo 'manylinux1_compatible = True' > /usr/local/lib/python3.7/_manylinux.py &&\
pip3 install orjson &&\
rm /usr/local/lib/python3.7/_manylinux.py

Upvotes: -3

Related Questions