Reputation: 29

How do I run an Ansible pip task that uses a requirements file over a proxy?

We have recently setup up a development box that we want to be able to install our software on. We are using Ansible and it all works fine on our local environments but this new box is set up to access the outside world via a proxy. This has meant that to get all the pip tasks working in ansible I have needed to set the environment on each of the them detailing the proxy details (as per http://docs.ansible.com/ansible/latest/user_guide/playbooks_environment.html).

- name: upgrade pip
  pip:
    virtualenv: "{{ wcsftp_sources_dir }}/{{ wcsftp_rev }}"
    virtualenv_command: virtualenv-2.7
    name: pip
    version: "{{ pip_version }}"
    umask: '0022'
  environment:
    http_proxy: http://proxy.example.com:8080
    https_proxy: http://proxy.example.com:8080
  tags:
    - dependencies
    - virtualenv

That has worked fine but when I am running a pip task that uses a requirements file I am getting errors that I am assuming is down to the proxy details not being passed into the call for installing each of the required pieces of software.

- name: create virtualenv for wcsftp
  pip:
    virtualenv: "{{ wcsftp_sources_dir }}/{{ wcsftp_rev }}"
    virtualenv_command: virtualenv-2.7
    requirements: "{{ wcsftp_sources_dir }}/{{ wcsftp_rev }}/wcsftp/requirements.txt"
    umask: '0022'
  environment:
    GIT_AUTH: "{{ gituser }}:{{ gitpass }}"
    GIT_ASKPASS: /usr/local/bin/git_env_askpass.py
    http_proxy: http://proxy.example.com:8080
    https_proxy: http://proxy.example.com:8080
    HTTP_PROXY: http://proxy.example.com:8080
    HTTPS_PROXY: http://proxy.example.com:8080
  tags:
    - dependencies
    - virtualenv

Results in the following errors:

build   10-Apr-2018 10:14:56    TASK [create virtualenv for wcsftp] ********************************************
build   10-Apr-2018 10:15:00    fatal: [localhost]: FAILED! => {"changed": false, "cmd": "/srv/wcsftp/master/bin/pip2 install -r /srv/wcsftp/master/wcsftp/requirements.txt", "failed": true, "msg": "stdout: Collecting git+https://github.com/prometheus/client_python.git (from -r /srv/wcsftp/master/wcsftp/requirements.txt (line 15))
Cloning https://github.com/prometheus/client_python.git to ./pip-VBNsmQ-build\nCollecting Twisted==17.5.0 (from -r /srv/wcsftp/master/wcsftp/requirements.txt (line 1))
Downloading Twisted-17.5.0.tar.bz2 (3.0MB)
Complete output from command python setup.py egg_info:
Download error on https://pypi.python.org/simple/incremental/: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:579) -- Some packages may not be found!
Couldn't find index page for 'incremental' (maybe misspelled?)
Download error on https://pypi.python.org/simple/: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:579) -- Some packages may not be found!
No local packages or download links found for incremental>=16.10.1
Traceback (most recent call last):
File \"<string>\", line 1, in <module>
File \"/tmp/pip-build-HAEyHG/Twisted/setup.py\", line 21, in <module>
setuptools.setup(**_setup[\"getSetupArgs\"]())
File \"/usr/lib64/python2.7/distutils/core.py\", line 112, in setup
_setup_distribution = dist = klass(attrs)
File \"/srv/wcsftp/master/lib/python2.7/site-packages/setuptools/dist.py\", line 265, in __init__
self.fetch_build_eggs(attrs.pop('setup_requires'))
File \"/srv/wcsftp/master/lib/python2.7/site-packages/setuptools/dist.py\", line 289, in fetch_build_eggs
parse_requirements(requires), installer=self.fetch_build_egg
File \"/srv/wcsftp/master/lib/python2.7/site-packages/pkg_resources.py\", line 618, in resolve
dist = best[req.key] = env.best_match(req, self, installer)
File \"/srv/wcsftp/master/lib/python2.7/site-packages/pkg_resources.py\", line 862, in best_match
return self.obtain(req, installer) # try and download/install
File \"/srv/wcsftp/master/lib/python2.7/site-packages/pkg_resources.py\", line 874, in obtain
return installer(requirement)
File \"/srv/wcsftp/master/lib/python2.7/site-packages/setuptools/dist.py\", line 339, in fetch_build_egg
return cmd.easy_install(req)
File \"/srv/wcsftp/master/lib/python2.7/site-packages/setuptools/command/easy_install.py\", line 617, in easy_install
raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('incremental>=16.10.1')
----------------------------------------
:stderr: Command \"python setup.py egg_info\" failed with error code 1 in /tmp/pip-build-HAEyHG/Twisted/

If I run the following command manually on the box then everything installs fine, including incremental:

pip install twisted==17.5.0 --proxy=http://proxy.example.com:8080

If I run ansible again after maunally intalling twisted I get the exact same SSL: UNKNOWN_PROTOCOL error as detailed above but this time it uses a cached version of Twisted-17.5.0.tar.bz2.

I have tried manually running the command and specifying the proxy information using both pip and pip2:

/srv/wcsftp/master/bin/pip2 install -r /srv/wcsftp/master/wcsftp/requirements.txt --proxy=http://webproxy.e.corp.services:80

and get a different error but it is still related to being unable to access the external site to download the incremental package:

Collecting Twisted==17.5.0 (from -r /srv/wcsftp/master/wcsftp/requirements.txt (line 1))
  Using cached Twisted-17.5.0.tar.bz2
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.python.org/simple/incremental/: [Errno 101] Network is unreachable -- Some packages may not be found!
    Couldn't find index page for 'incremental' (maybe misspelled?)
    Download error on https://pypi.python.org/simple/: [Errno 101] Network is unreachable -- Some packages may not be found!
    No local packages or download links found for incremental>=16.10.1

Upvotes: 3

How do I run an Ansible pip task that uses a requirements file over a proxy?

Answers (2)

Related Questions