CODEWITHSUNDEEP
pythongoogle-cloud-dataflowapache-beamorjson
Prasad Sawant
Prasad Sawant

Reputation: 205

Apache beam(2.32.0) installation fails

We have some issues deploying our GCP Dataflow pipeline. After some analysis, found that the latest version of apache-beam has some issues while installing. To replicate the issue I created a virtualenv and ran the below

pip install apache-beam==2.32.0

Below errors started to pop while installing the 'orjson' dependency,

  Using cached orjson-3.6.3.tar.gz (548 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: 'c:\temp\virtu\scripts\python.exe' 'c:\temp\virtu\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py' prepare_metadata_for_build_wheel 'C:\Users\prasasaw\AppData\Local\Temp\tmpmekx1jjj'
         cwd: C:\Users\prasasaw\AppData\Local\Temp\pip-install-0yofoe55\orjson_62f1ca2674934a7f8c45b08e87e05a4b
    Complete output (6 lines):

    Cargo, the Rust package manager, is not installed or is not on PATH.
    This package requires Rust and Cargo to compile extensions. Install it through
    the system's package manager or via https://rustup.rs/

Note that the previous versions of Apache beam like 2.30.0 does not have the dependency on 'orjson' and it works just fine when you do

pip install apache-beam==2.30.0

I tried to install RUST but it failed for some 'pysam' dependency. So would like to know what is the correct way to install the 'orjson' dependency

I saw this GitHub issue for orjson installation but could not find much from it. https://github.com/readthedocs/readthedocs.org/issues/7687

//Prasad.

Upvotes: 1

Views: 2316

Answers (3)

CaptainNabla
CaptainNabla

Reputation: 1166

I'm a bit late to the party, but I ran into this issue as well today. I solved it by switching to a 64-bit Python environment (I accidently installed 32-bit).

Upvotes: 1

alxrsngrtn
alxrsngrtn

Reputation: 1

According to the orjson docs, you need to upgrade pip to be above 20.3:

pip install --upgrade "pip>=20.3" # manylinux_x_y, universal2 wheel support
pip install --upgrade orjson

(docs)

Upvotes: 0

Andy Xu
Andy Xu

Reputation: 101

The orjson dependency was introduced in https://github.com/apache/beam/pull/14690/files. According to the comment: orjson, only available on Python 3.6 and above. You may want to check your python version.

Upvotes: 0

Related Questions