Issues with pipenv install on M1 Mac for python dependencies

Question

I'm trying to set up a local repository that was originally created on intel-based macs. The repository uses pyenv and pipenv to manage python dependencies. Whenever I try to run pipenv install, I get the following error:

ERROR:pip.subprocessor:[present-rich] pip subprocess to install build dependencies exited with 1
[ResolutionFailure]:   File "/Users/xur19/.pyenv/versions/3.8.10/lib/python3.8/site-packages/pipenv/resolver.py", line 782, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "/Users/xur19/.pyenv/versions/3.8.10/lib/python3.8/site-packages/pipenv/resolver.py", line 730, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/Users/xur19/.pyenv/versions/3.8.10/lib/python3.8/site-packages/pipenv/resolver.py", line 709, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/Users/xur19/.pyenv/versions/3.8.10/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 1105, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/Users/xur19/.pyenv/versions/3.8.10/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 894, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/Users/xur19/.pyenv/versions/3.8.10/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 689, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: pip subprocess to install build dependencies exited with 1

The repository has an existing Pipfile and Pipfile.lock

My steps for setup are:

brew install pyenv
pyenv install 3.8.7
pyenv global 3.8.7
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init)"\nfi' >> ~/.zshrc
brew install python
pip install pipenv
pipenv shell
pipenv install

My Mac is an M1 mac and I've tried the following:

• Try to make the repo work with ARM architecture. I've tried using python 3.8.10 instead of 3.8.7 since that is the latest version that supports the M1 chip, and I've tried removing some dependencies that aren't being used that don't support M1 (cx-oracle). I've also tried running pipenv lock --pre and pipenv install --skip-lock but both of those result in the same error. I've also tried deleting my Pipfile.lock and only using the Pipfile, but that also fails.
• Try to make the repo work with Rosetta 2. I've tried installing homebrew with Rosetta (to /usr/local/bin/brew) and I've used the arch -x86_64 prefixes with brew (with a separate Rosetta enabled terminal, changed my paths, etc). When I try that, I get similar errors. I've reviewed several stackoverflow and medium articles that describe how to install python/pipenv/pyenv with Rosetta, but I'm unable to get past this error.

I'm not sure what else to try - was wondering if anyone had advice on how to tackle this. For context, here is my Pipfile.

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
autopep8 = "*"
flake8 = "*"
black = "==20.8b1"

[packages]
celery = "==4.4.7"
channels = "==3.0.3"
channels-redis = "==3.2.0"
#cx_Oracle = "==8.2.1" # Django DATABASE
Django = "==2.2.24"
django-allauth = "==0.44.0"
django-celery-beat = "==2.2.0"
django-cors-headers = "==3.7.0"
django-enumfield = "==1.5"
django-environ = "==0.4.5"
django-extensions = "==3.1.2"
django-filter = "==2.1.0"
django-graphql-jwt = "==0.3.1"
django-graphql-persist = "==0.1.1"
django-jet = "==1.0.8"
django-redis = "==4.12.1"
django-reversion = "==3.0.9"
django-reversion-compare = "==0.14.0"
django-measurement = "==3.2.3"
django-jsoneditor = "==0.1.6"
django-storages = {extras = ["google"],version = "==1.11.1"}
flower = "==0.9.7"
future = "==0.18.2"
google-auth = "==1.29.0"
google-auth-oauthlib = "==0.4.4"
google-cloud-storage = "==1.37.1"
google-cloud-pubsub = "==2.4.0"
graphene = "==2.1.7"
graphql-core = "==2.2"
graphene-django = "==2.7"
graphene-file-upload = "==1.3.0"
graphql-relay = "==2"
graphql-server-core = "==1.1.1"
gunicorn = "==19.9.0"
ijson = "==3.1.4"
ldap3 = "==2.9"
matplotlib = "==3.4.1"
munch = "==2.5.0"
numpy = "==1.20.2"
openpyxl = "==3.0.7"
pandas = "==1.2.4"
pillow = "==8.2.0"
psycopg2-binary = "==2.8.6"
pyhumps = "==1.6.1"
pyjwt = "==1.7.1"
pymongo = "==3.11.3"
pyteomics = "==4.4.2"
python-pptx = "==0.6.18"
pytz = "==2018.9"
redis = "==3.2.0" # Doesn't get installed by django-redis
requests = "==2.21.0"
tqdm = "==4.60.0"
xlrd = "==2.0.1"
jinja2 = "==3.0.2"
premailer = "==3.10.0"
docxtpl = "==0.14.2"
inflection = "==0.4.0"
sympy = "==1.9"

[requires]
python_version = "3.7"

Answer 1

In my case I did the following:

1. arch -arm64 brew install libffi
2. export LDFLAGS="-L/opt/homebrew/opt/libffi/lib"
3. export CPPFLAGS="-I/opt/homebrew/opt/libffi/include"
4. CONFIGURE_OPTS='--with-system-ffi' pyenv install 3.7.10 pyenv local 3.7.10
5. pipenv install

And then it worked!

Answer 2

I'm not 100% sure, since I was dealing with 3.7.10, but my problem looked suspiciously similar. Based on this issue on pipenv, I was led to this article, which led me to a solution.

The summarized solution from that article is as follows.

I'm not sure you need to install libffi from brew and make these exports, but I was in a morass so it may help.

brew install libffi
export LDFLAGS="-L/opt/homebrew/opt/libffi/lib"
export CPPFLAGS="-I/opt/homebrew/opt/libffi/include"

This part you definitely need to do.

CONFIGURE_OPTS='--with-system-ffi' pyenv install 3.7.10
pyenv local 3.7.10

Then edit the file that causes the MemoryError. For 3.7.10 ~/.pyenv/versions/3.7.10/lib/python3.7/ctypes/__init__.py. Comment or remove the last line of this function.

def _reset_cache():
    _pointer_type_cache.clear()
    _c_functype_cache.clear()
    if _os.name == "nt":
        _win_functype_cache.clear()
    # _SimpleCData.c_wchar_p_from_param
    POINTER(c_wchar).from_param = c_wchar_p.from_param
    # _SimpleCData.c_char_p_from_param
    POINTER(c_char).from_param = c_char_p.from_param
    _pointer_type_cache[None] = c_void_p
    # XXX for whatever reasons, creating the first instance of a callback
    # function is needed for the unittests on Win64 to succeed.  This MAY
    # be a compiler bug, since the problem occurs only when _ctypes is
    # compiled with the MS SDK compiler.  Or an uninitialized variable?
    CFUNCTYPE(c_int)(lambda: None) # comment or remove me

Then it worked for me. pipenv installed many things without issue.