Unable to install pycurl==7.43.0.5 on AWS Elastic Beanstalk

Question

Hi guys I have been trying to add pycurl on EB running with python 3.7 however when I try to launch my celery app it returns the following errors:

"The curl client requires the pycurl library"

the command that is being executed is:

celery -A sportspot worker -B

[2020-06-23 07:36:14,815: CRITICAL/MainProcess] Unrecoverable error: ImportError('The curl client requires the pycurl library.')
2020-06-23 07:36:16,508 P8649 [INFO]    Traceback (most recent call last):
2020-06-23 07:36:16,508 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/kombu/asynchronous/http/__init__.py", line 20, in get_client
2020-06-23 07:36:16,508 P8649 [INFO]        return hub._current_http_client
2020-06-23 07:36:16,508 P8649 [INFO]    AttributeError: 'Hub' object has no attribute '_current_http_client'
2020-06-23 07:36:16,508 P8649 [INFO]    
2020-06-23 07:36:16,508 P8649 [INFO]    During handling of the above exception, another exception occurred:
2020-06-23 07:36:16,508 P8649 [INFO]    
2020-06-23 07:36:16,508 P8649 [INFO]    Traceback (most recent call last):
2020-06-23 07:36:16,508 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/celery/worker/worker.py", line 208, in start
2020-06-23 07:36:16,508 P8649 [INFO]        self.blueprint.start(self)
2020-06-23 07:36:16,508 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
2020-06-23 07:36:16,508 P8649 [INFO]        step.start(parent)
2020-06-23 07:36:16,508 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
2020-06-23 07:36:16,508 P8649 [INFO]        return self.obj.start()
2020-06-23 07:36:16,508 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
2020-06-23 07:36:16,509 P8649 [INFO]        blueprint.start(self)
2020-06-23 07:36:16,509 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
2020-06-23 07:36:16,509 P8649 [INFO]        step.start(parent)
2020-06-23 07:36:16,509 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 599, in start
2020-06-23 07:36:16,509 P8649 [INFO]        c.loop(*c.loop_args())
2020-06-23 07:36:16,509 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/celery/worker/loops.py", line 83, in asynloop
2020-06-23 07:36:16,509 P8649 [INFO]        next(loop)
2020-06-23 07:36:16,509 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 301, in create_loop
2020-06-23 07:36:16,509 P8649 [INFO]        item()
2020-06-23 07:36:16,509 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/vine/promises.py", line 170, in __call__
2020-06-23 07:36:16,509 P8649 [INFO]        return self.throw()
2020-06-23 07:36:16,509 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/vine/promises.py", line 167, in __call__
2020-06-23 07:36:16,509 P8649 [INFO]        retval = fun(*final_args, **final_kwargs)
2020-06-23 07:36:16,509 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/kombu/transport/SQS.py", line 391, in _schedule_queue
2020-06-23 07:36:16,509 P8649 [INFO]        queue, callback=promise(self._loop1, (queue,)),
2020-06-23 07:36:16,509 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/kombu/transport/SQS.py", line 407, in _get_bulk_async
2020-06-23 07:36:16,510 P8649 [INFO]        return self._get_async(queue, maxcount, callback=callback)
2020-06-23 07:36:16,510 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/kombu/transport/SQS.py", line 417, in _get_async
2020-06-23 07:36:16,510 P8649 [INFO]        qname, count=count, connection=self.asynsqs(queue=qname),
2020-06-23 07:36:16,510 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/kombu/transport/SQS.py", line 571, in asynsqs
2020-06-23 07:36:16,510 P8649 [INFO]        region=self.region
2020-06-23 07:36:16,510 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/kombu/asynchronous/aws/sqs/connection.py", line 27, in __init__
2020-06-23 07:36:16,510 P8649 [INFO]        **kwargs
2020-06-23 07:36:16,510 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/kombu/asynchronous/aws/connection.py", line 198, in __init__
2020-06-23 07:36:16,510 P8649 [INFO]        **http_client_params)
2020-06-23 07:36:16,510 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/kombu/asynchronous/aws/connection.py", line 151, in __init__
2020-06-23 07:36:16,510 P8649 [INFO]        self._httpclient = http_client or get_client()
2020-06-23 07:36:16,510 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/kombu/asynchronous/http/__init__.py", line 22, in get_client
2020-06-23 07:36:16,510 P8649 [INFO]        client = hub._current_http_client = Client(hub, **kwargs)
2020-06-23 07:36:16,510 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/kombu/asynchronous/http/__init__.py", line 13, in Client
2020-06-23 07:36:16,510 P8649 [INFO]        return CurlClient(hub, **kwargs)
2020-06-23 07:36:16,511 P8649 [INFO]      File "/var/app/venv/staging-LQM1lest/lib/python3.7/site-packages/kombu/asynchronous/http/curl.py", line 43, in __init__
2020-06-23 07:36:16,511 P8649 [INFO]        raise ImportError('The curl client requires the pycurl library.')
2020-06-23 07:36:16,511 P8649 [INFO]    ImportError: The curl client requires the pycurl library.
2020-06-23 07:36:16,511 P8649 [INFO]     
2020-06-23 07:36:16,511 P8649 [INFO]     -------------- celery@ip-172-31-17-183.ec2.internal v4.4.5 (cliffs)
2020-06-23 07:36:16,511 P8649 [INFO]    --- ***** ----- 
2020-06-23 07:36:16,511 P8649 [INFO]    -- ******* ---- Linux-4.14.177-139.254.amzn2.x86_64-x86_64-with-glibc2.2.5 2020-06-23 07:36:14
2020-06-23 07:36:16,511 P8649 [INFO]    - *** --- * --- 
2020-06-23 07:36:16,511 P8649 [INFO]    - ** ---------- [config]
2020-06-23 07:36:16,511 P8649 [INFO]    - ** ---------- .> app:         sportspot:0x7f1916c1c290
2020-06-23 07:36:16,511 P8649 [INFO]    - ** ---------- .> transport:   sqs://localhost//
2020-06-23 07:36:16,511 P8649 [INFO]    - ** ---------- .> results:     disabled://
2020-06-23 07:36:16,511 P8649 [INFO]    - *** --- * --- .> concurrency: 1 (prefork)
2020-06-23 07:36:16,511 P8649 [INFO]    -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
2020-06-23 07:36:16,511 P8649 [INFO]    --- ***** ----- 
2020-06-23 07:36:16,511 P8649 [INFO]     -------------- [queues]
2020-06-23 07:36:16,512 P8649 [INFO]                    .> celery           exchange=celery(direct) key=celery

however I have tried different workarounds to address the problem with no luck, some of them like specifying in my requierements.txt different variants of the pycurl like the following (of course I deploy with only one option at a time not all of them):

pycurl==7.43.0.5 #no luck here
pycurl==7.43.0.5 --global-option="--with-nss" #no luck here
pycurl==7.43.0.5 --global-option="--with-openssl" #no luck here
pycurl==7.19.3 #no luck dowgrading

I also tried installing the packages from the eb packages commands like so:

packages:
    yum:
        libcurl-devel: []
        libcurl4-openssl-dev: []
        libssl-dev: []
        python-pycurl: [7.19.3]

if I ssh into the eb instance then go to the venv and import the library I get the following

[ec2-user@ip-172-31-17-183 ~]$ source /var/app/venv/staging-LQM1lest/bin/activate
(staging) [ec2-user@ip-172-31-17-183 ~]$ python3 -c 'import pycurl'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)
(staging) [ec2-user@ip-172-31-17-183 ~]$ python -c 'import pycurl'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)
(staging) [ec2-user@ip-172-31-17-183 ~]$ 

Also for some reason eb does not store the virtual environment on the classic opt/python path so it made stuff even more complicated from the beginning, since literally there is 0 documentation about it. every path was different for example migrating or collecting static files, I had to change the path in order to activate the virtual environment.

all tutorials use the old "opt" path even on amazon's documentation...

At this point I seriously don't know what else to try since this works perfectly when I execute on localhost.

I have managed to deploy this without celery but it is really important for us to execute that part too.

if you read the entire question I really appreciate the time you took to analyze this! :)

Answer 1

I managed to fix this problem for an EB instance running on python 3.7 by doing the following:

first I ssh into the instance, then I ran the following commands:

sudo su
cd /var/app/
source /venv/staging-LQM1lest/bin/activate
(staging): pip uninstall pycurl
(staging): export CPPFLAGS=-I/usr/local/opt/openssl/include
(staging): export LDFLAGS=-L/usr/local/opt/openssl/lib
(staging): pip install pycurl --global-option="--with-openssl"

since this commands worked perfectly I converted them into a single command on the ebconfig files like so:

first remove pycurl from the requirements.txt then add the following to your eb extensions file:

container_commands:
  .
  . some other commands...
  .
  04_installpycurl:
    command: "source /var/app/venv/staging-LQM1lest/bin/activate && export CPPFLAGS=-I/usr/local/opt/openssl/include && export LDFLAGS=-L/usr/local/opt/openssl/lib && pip install pycurl --global-option='--with-openssl'"

Notice that I tried to install pycurl --global-option='--with-openssl' from the requirements.txt file with no results, it seems like exporting those two variables make a difference on the installation, I have no idea as to why or what those things do but that fixed the problem :)

finally notice that my command first goes to the venv and activates it so if your path to your venv is different you must change that!