Ansible aws_s3 module fails says Boto3 is missing when it is not

Question

This Play installs python3, pip3, boto3 and botocore, and tries to use aws_s3 module to download a file:

TASK [run yum update -y using yum module] 
**********************************************************************
ok: [ip-10-200-2-137.us-west-2.compute.internal]

TASK [Install python3 and pip3] *************************************************************************************************
changed: [ip-10-200-2-137.us-west-2.compute.internal]

TASK [Install boto3 and botocore with pip3 module] ******************************************************************************
changed: [ip-10-200-2-137.us-west-2.compute.internal]

TASK [Create a directory if it does not exist using file module] ****************************************************************
changed: [ip-10-200-2-137.us-west-2.compute.internal]

TASK [downlod file from s3 with aws_s3 module] **********************************************************************************
fatal: [ip-10-200-2-137.us-west-2.compute.internal]: FAILED! => 
{"changed": false, "msg": "Python modules \"botocore\" or \"boto3\" 
are missing, please install both"}

It fails because it says boto3 is missing, but it actually it is not:

From the Target host you can see that boto3 was installed:

[ec2-user@ip-10-200-2-137 ~]$ pip3 freeze
boto3==1.9.120
botocore==1.12.120
docutils==0.14
jmespath==0.9.4
python-dateutil==2.8.0
s3transfer==0.2.0
six==1.12.0
urllib3==1.24.1
[ec2-user@ip-10-200-2-137 ~]

This is the task that installed boto3:

- name: Install boto3 and botocore with pip3 module
    pip:
      name: 
      - boto3
      - botocore
      executable: pip-3.7

This is the task that fails:

- name: downlod file from s3 with aws_s3 module 
    aws_s3:
      bucket: mybucket
      object: mybucket/jre-8u201-linux-x64.tar.gz
      dest: /home/ec2-user/updater/jre-8u201-linux-x64.tar.gz
      mode: get   

The target host does have two versions of Python installed:

[ec2-user@ip-10-200-2-157 ~]$ which python
/usr/bin/python
[ec2-user@ip-10-200-2-157 ~]$ which python3
/usr/bin/python3

My config file looks like this:

[defaults]
private_key_file=/home/ec2-user/manual-builds/key.pem
ansible_python_interpreter=/usr/bin/python3

Is this a bug? I see some similar questions have been asked going back almost a year, but I see no solutions - thanks much for any help.

Answer 1

One other situation that can cause this is that the task installing the library is doing it as a non-root user like this:

- name:           Ensure botocore and boto3 modules are installed
  pip:
    name:         [ "boto3", "botocore"]
    extra_args:   "--user"

But then your task that requires the module has become: true

In that case, you have a couple of potential solutions:

1. Install your modules as global modules (ensuring they are always available)
2. Install your modules as user libraries with become: true (ensuring they are available to the super user)
3. Install your modules as user libraries in both the connecting user's profile and that of the super user. (ensuring they are available to both the super user and the connecting user)

Of course, the same caveats apply as the other answers - there can be multiple instances of python installed.

Answer 2

I hit this error because on my EC2 system, there happened to be two versions of Python installed. /usr/bin/python3 and /usr/local/bin/python3. Only the latter had pip installed at all, under /usr/local/bin/pip3. So by default, somehow, Ansible was attempting to use the Python interpretter /usr/bin/python3, but this Python does not have any pip and does not have boto3 or other required libraries. However, the Ansible module for pip seemed to be resolving to /usr/local/bin/pip3 despite the fact that this did not match the Python interpreter. So when I include a task like this

  - name: Ensure botocore and boto3 modules are installed
    pip: 
      name: [ "boto3", "botocore"]
      extra_args: "--user"

Ansible returns "OK" because the pip it uses does indeed detect boto3 and botocore. Ansible is not aware that these packages are installed under the "wrong" Python which does not actually match the Python at /usr/bin/python3 which it was attempting to use for things like S3 object actions.

I think I have resolved this by including this in my playbook

- name: Run AWS Tasks
  hosts: "{{ myhosts | default('all') }}" # read from cli a single host or list of hosts
  gather_facts: yes
  vars:
    ansible_python_interpreter: /usr/local/bin/python3 

which forces Ansible to use the Python interpreter at /usr/local/bin/python3 instead of the one it mistakenly auto-detects at /usr/bin/python3.

Note that I think its somewhat precarious, and possibly dangerous, to attempt to install libraries for the Python at /usr/bin/python3 since I believe this is the "system Python". I think this is how I ended up with a separate Python at /usr/local/bin/python3 in the first place.

Also note that its helpful to debug this with a playbook task that can print out some of the internal Ansible variables to help reveal which Python executable is being used such as

  - name: Check Python version
    debug:
      msg:
        - "Python interpreter: {{ ansible_facts.python.executable }}"

Answer 3

https://github.com/aws/aws-cli/issues/3092#issuecomment-550281243

Uninstalling python3-botocore then installing with pip3 seems to work.

Answer 4

Ansible use /usr/bin/python as default python interpreter. And you install AWS libraries for python3 only:

- name: Install boto3 and botocore with pip3 module
    pip:
      name: 
      - boto3
      - botocore
      executable: pip-3.7

You can install AWS libraries to python2 by using pip or install for both (python3 and python2) or you can define: ansible_python_interpreter=/usr/bin/python3 in your inventory file then you limit ansible execution to python3 only.

Answer 5

Ansible might not be referring to the incorrect python version. Simply run ansible version command to see where ansible is pointing to.

ansible --version
ansible 2.9.9
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.17 (default, Apr 15 2020, 17:20:14) [GCC 7.5.0]

As you see Ansibe is pointing to the Python module located at /usr/lib/python2.7/ (ansible python module location) you can see if this python version has boto3 and botocore packages are installed in /usr/lib/python2.7/dist-packages/ path.

If it's missing just install boto3 and botocore there.

Answer 6

When the ansible says

Failed to import the required Python library (botocore or boto3) Please read the module documentation and install it in the appropriate location.

Modules might already be installed. Confirm by python -m pip list installed command. Then open python in question directly in a terminal by typing python and try to import boto3 or botocore by import boto3. if it throws any error try to work out the error.

This answer might be useful.

Answer 7

The problem was that my playbook had two tasks and Ansible was using the python2 interpreter for the first one AND the second one. The second task needed the python3 interpreter to work so I had to specifiy it at the task level:

- name: downlod file from s3 with aws_s3 module
  vars:
      ansible_python_interpreter: /usr/bin/python3    
  aws_s3:
      bucket: launch-data
      object: jre-8u201-linux-x64.tar.gz
      dest: /home/ec2-user/updater/jre-8u201-linux-x64.tar.gz
      mode: get