CODEWITHSUNDEEP
amazon-web-servicesdockerboto3aws-cli
mrc
mrc

Reputation: 3153

Why am I getting the error The source profile "default" must have credentials using Boto3 in Docker?

I have a dockerfile:

FROM apache/airflow:2.1.2-python3.8

ENV PYTHONPATH "${PYTHONPATH}:/"

ADD ./aws/credentials /home/airflow/.aws/credentials
ADD ./aws/config /home/airflow/.aws/config

RUN pip install -r requirements.pip

And a docker-compose.yml:

version: '3'
services:
  webserver:
    image: airflow2
    command: webserver
    ports:
      - 8080:8080
    healthcheck:
      test: [ "CMD", "curl", "--fail", "http://localhost:8080/health" ]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always
    build:
      context: .
      dockerfile: Dockerfile
    env_file:
      - ./airflow.env
    container_name: webserver
    volumes:
      - ./database_utils:/database_utils
  scheduler:
    image: sch-airflow2
    command: scheduler
    healthcheck:
      test: [ "CMD-SHELL", 'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"' ]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always
    container_name: scheduler
    build:
      context: .
      dockerfile: Dockerfile
    env_file:
      - ./airflow.env
    volumes:
      volumes:
      - ./database_utils:/database_utils
    depends_on:
      - webserver

When running an script that uses boto3 to connect to s3 it works fine. The issue here is that I am adding the credentials to the image which is a bad practice. Then I delete this line from the dockerfile:

ADD ./aws/credentials /home/airflow/.aws/credentials

And I add this in the docker-compose volume section:

- ./aws:/home/airflow/.aws

If I run the script now it fails with:

botocore.exceptions.InvalidConfigError: The source profile "default" must have credentials.

I understant the error and I checked that credentials are provided in the docker container:

aws configure list

enter image description here

It seems fine, so I don't get it why it returns the error. I also tried setting up the ENV variables in the ./airflow.env:

AWS_ACCESS_KEY_ID=XXXXXXXXX
AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXX

The only way that I achieve it working is with the ADD in the dockerfile.

Upvotes: 1

Views: 12082

Answers (1)

Ermiya Eskandary
Ermiya Eskandary

Reputation: 23612

Something somewhere is specifying the default profile to be used.

This can happen because:

  • You specify --profile default as a command line parameter
  • You have an environment variable of AWS_PROFILE with value of default
  • Your ~/.aws/credentials contains [default]
  • Your ~/.aws/config file contains [default]

The above, ordered in terms of precedence, tell AWS to use the default profile.

Double-check the default profile is not being referenced in any of the above (most likely your config or credentials file).

Upvotes: 3

Related Questions