How to use the AWS Python SDK while connecting via SSO credentials

Question

I am attempting to create a python script to connect to and interact with my AWS account. I was reading up on it here https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html

and I see that it reads your credentials from ~/.aws/credentials (on a Linux machine). I however and not connecting with an IAM user but SSO user. Thus, the profile connection data I use is located at ~/.aws/sso/cache directory.

Inside that directory, I see two json files. One has the following keys:

• startUrl
• region
• accessToken
• expiresAt

the second has the following keys:

• clientId
• clientSecret
• expiresAt

I don't see anywhere in the docs about how to tell it to use my SSO user.

Thus, when I try to run my script, I get error such as

botocore.exceptions.ClientError: An error occurred (AuthFailure) when calling the DescribeSecurityGroups operation: AWS was not able to validate the provided access credentials

even though I can run the same command fine from the command prompt.

Answer 1

Many other applications don't consider sso credentials, at least not by default (for example java aws sdks). I found it easiest to just create a valid ~/.aws/credentials file from the sso credentials. Here is a simple script that will do this:

# get valid sso creds
aws sso login --sso-session awsaccess

# location for the final credentials file
creds_file="$HOME/.aws/credentials"

# new temp file that we will build below
temp_file="$HOME/.aws/tmpcreds.txt"
rm -rf $temp_file

# get the names of all your configured profiles using aws configure list-profiles.
for profile in $(aws configure list-profiles); do
# for each profile, get the corresponding credentials
  creds=$(aws configure export-credentials --profile $profile)
# parse the credentials output into the proper format for the ~/.aws/credentials file, and append to the temp file.
  printf "[%s]\naws_access_key_id=%s\naws_secret_access_key=%s\naws_session_token=%s\n\n" \
    "$profile" \
    "$(jq -r .AccessKeyId <<< $creds)" \
    "$(jq -r .SecretAccessKey <<< $creds)" \
    "$(jq -r .SessionToken <<< $creds)" \
    >> $temp_file
done


# now the temp file has all of your configured profiles and credentials in the proper format. replace the actual creds file with this one
mv $temp_file $creds_file

Answer 2

A really simple solution for refreshing the SSO access is using subprocess to run the AWS command:

def refresh_sso_access():
    print("Refreshing AWS SSO access..")
    profiles = boto3.session.Session().available_profiles
    profile = "dev" if "dev" in profiles else profiles[0]
    subprocess.run(["aws", "sso", "login", "--profile", profile])

This will open the browser window for you to refresh the access.

Answer 3

If you have ever executed aws sso configure command in the terminal before and followed the prompts, then you should have something like the following in your ~/.aws/config file.

[profile <profile_name>]
sso_session = <sso_session_name>
sso_account_id = <account_id>
sso_role_name = <role_name>
region = <region>
output = text
[sso-session <sso_session_name>]
sso_start_url = <sso_start_url>
sso_region = <region>
sso_registration_scopes = sso:account:access

After logging in the terminal using aws sso login --profile <profile_name> where <profile_name> must match one of the profiles in the ~/.aws/config file above, you need to set AWS_PROFILE environment variable to the same profile name above. If you want to set the environment variable only for the current terminal (where you run your Python script), run export AWS_PROFILE=<profile_name> in the terminal. Or if it is a profile that you often use, add the same export statement above to something like ~/.bashrc file. The good thing about setting the environment variable is that you do not need to edit your code.

Answer 4

UPDATED for latest boto3 on 2023.10.23 (hat tip commenter @Adam Smith whose invisible hand guided us to updating extracting the role credentials in newer versions of boto3):

So here's the long and hairy answer tested on boto3==1.28.69:

It's an eight-step process where:

1. register the client using sso-oidc.register_client
2. start the device authorization flow using sso-oidc.start_device_authorization
3. redirect the user to the sso login page using webbrowser.open
4. poll sso-oidc.create_token until the user completes the signin
5. list and present the account roles to the user using sso.list_account_roles
6. get role credentials using sso.get_role_credentials
7. create a new boto3 session with the session credentials from (6)
8. eat a cookie

Step 8 is really key and should not be overlooked as part of any successful authorization flow.

In the sample below the account_id should be the account id of the account you are trying to get credentials for. And the start_url should be the url that aws generates for you to start the sso flow (in the AWS SSO management console, under Settings).

from time import time, sleep
import webbrowser
from boto3.session import Session

# if your sso is setup in a different region, you will
# want to include region_name=sso_region in the 
# session constructor below
session = Session()
account_id = '1234567890'
start_url = 'https://d-0987654321.awsapps.com/start'
region = 'us-east-1' 
sso_oidc = session.client('sso-oidc')
client_creds = sso_oidc.register_client(
    clientName='myapp',
    clientType='public',
)
device_authorization = sso_oidc.start_device_authorization(
    clientId=client_creds['clientId'],
    clientSecret=client_creds['clientSecret'],
    startUrl=start_url,
)
url = device_authorization['verificationUriComplete']
device_code = device_authorization['deviceCode']
expires_in = device_authorization['expiresIn']
interval = device_authorization['interval']
webbrowser.open(url, autoraise=True)
for n in range(1, expires_in // interval + 1):
    sleep(interval)
    try:
        token = sso_oidc.create_token(
            grantType='urn:ietf:params:oauth:grant-type:device_code',
            deviceCode=device_code,
            clientId=client_creds['clientId'],
            clientSecret=client_creds['clientSecret'],
        )
        break
    except sso_oidc.exceptions.AuthorizationPendingException:
        pass
 
access_token = token['accessToken']
sso = session.client('sso')
account_roles = sso.list_account_roles(
    accessToken=access_token,
    accountId=account_id,
)
roles = account_roles['roleList']
# simplifying here for illustrative purposes
role = roles[0]

# earlier versions of the sso api returned the 
# role credentials directly, but now they appear
# to be in a subkey called `roleCredentials`
role_creds = sso.get_role_credentials(
    roleName=role['roleName'],
    accountId=account_id,
    accessToken=access_token,
)['roleCredentials']
session = Session(
    region_name=region,
    aws_access_key_id=role_creds['accessKeyId'],
    aws_secret_access_key=role_creds['secretAccessKey'],
    aws_session_token=role_creds['sessionToken'],
)

Answer 5

What works for me is the following:

import boto 3


session = boto3.Session(profile_name="sso_profile_name")
session.resource("whatever")

using boto3==1.20.18.

This would work if you had previously configured SSO for aws ie. aws configure sso.

Interestingly enough, I don't have to go through this if I use ipython, I just aws sso login beforehand and then call boto3.Session(). I am trying to figure out whether there is something wrong with my approach - I fully agree with what was said above with respect to transparency and although it is a working solution, I am not in love with it.

---

EDIT: there was something wrong and here is how I fixed it:

1. run aws configure sso (as above);
2. install aws-vault - it basically replaces aws sso login --profile <profile-name>;
3. run aws-vault exec <profile-name> to create a sub-shell with AWS credentials exported to environment variables.

Doing so, it is possible to run any boto3 command both interactively (eg. iPython) and from a script, as in my case. Therefore, the snippet above simply becomes:

import boto 3


session = boto3.Session()
session.resource("whatever")

Here for further details on AWS vault.

Answer 6

This was fixed in boto3 1.14.

So given you have a profile like this in your ~/.aws/config:

[profile sso_profile]
sso_start_url = <sso-url>
sso_region = <sso-region>
sso_account_id = <account-id>
sso_role_name = <role>
region = <default region>
output = <default output (json or text)>

And then login with $ aws sso login --profile sso_profile

You will be able to create a session:

import boto3
boto3.setup_default_session(profile_name='sso_profile')
client = boto3.client('<whatever service you want>')

Answer 7

A well-formed boto3-based script should transparently authenticate based on profile name. It is not recommended to handle the cached files or keys or tokens yourself, since the official code methods might change in the future. To see the state of your profile(s), run aws configure list --examples:

$ aws configure list --profile=sso

      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                      sso           manual    --profile

The SSO session associated with this profile has expired or is otherwise invalid.
To refresh this SSO session run aws sso login with the corresponding profile.

$ aws configure list --profile=old

      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                      old           manual    --profile
access_key     ****************3DSx shared-credentials-file    
secret_key     ****************sX64 shared-credentials-file    
    region                us-west-1              env    ['AWS_REGION', 'AWS_DEFAULT_REGION']

Answer 8

Your current .aws/sso/cache folder structure looks like this:

$ ls
botocore-client-XXXXXXXX.json       cXXXXXXXXXXXXXXXXXXX.json

The 2 json files contain 3 different parameters that are useful.

botocore-client-XXXXXXXX.json -> clientId and clientSecret
cXXXXXXXXXXXXXXXXXXX.json -> accessToken

Using the access token in cXXXXXXXXXXXXXXXXXXX.json you can call get-role-credentials. The output from this command can be used to create a new session.

Your Python file should look something like this:

import json
import os
import boto3

dir = os.path.expanduser('~/.aws/sso/cache')

json_files = [pos_json for pos_json in os.listdir(dir) if pos_json.endswith('.json')]

for json_file in json_files :
    path = dir + '/' + json_file
    with open(path) as file :
        data = json.load(file)
        if 'accessToken' in data:
            accessToken = data['accessToken']

client = boto3.client('sso',region_name='us-east-1')
response = client.get_role_credentials(
    roleName='string',
    accountId='string',
    accessToken=accessToken
)

session = boto3.Session(aws_access_key_id=response['roleCredentials']['accessKeyId'], aws_secret_access_key=response['roleCredentials']['secretAccessKey'], aws_session_token=response['roleCredentials']['sessionToken'], region_name='us-east-1')