CODEWITHSUNDEEP
pythonamazon-web-servicesboto3aws-organizations
Geoff Sweet
Geoff Sweet

Reputation: 663

How do you use "NextToken" in AWS API calls

I've run into a little issue that I am really struggling to understand how it works. I have a tool I am writing that basically does a describe-organization to collect all the accounts in our AWS organization. Per the documentation here it says it responds with a json of the accounts which in my case will be hundreds and hundreds of accounts. So I wrote some very simple code to switch roles into our master account and make the call:

import boto3
import uuid
import pprint

iam_client = boto3.client('iam')
sts_client = boto3.client('sts')
org_client = boto3.client('organizations')


print("Starting in account: %s" % sts_client.get_caller_identity().get('Account'))

assumedRoleObject = sts_client.assume_role(
    RoleArn="arn:aws:iam::123456xxx:role/MsCrossAccountAccessRole",
    RoleSessionName="MasterPayer"
)

credentials = assumedRoleObject['Credentials']

org_client = boto3.client(
    'organizations',
    aws_access_key_id = credentials['AccessKeyId'],
    aws_secret_access_key = credentials['SecretAccessKey'],
    aws_session_token = credentials['SessionToken'],
)

getListAccounts = org_client.list_accounts(
    NextToken='string'
)

But when I execute the code, I get the following error:

"botocore.errorfactory.InvalidInputException: An error occurred (InvalidInputException) when calling the ListAccounts operation: You specified an invalid value for nextToken. You must get the value from the response to a previous call to the API."

I'm really stumped on what that means. I see the NextToken, and I can find many references to it in the AWS documentation but I can't figure out how to actually USE it. Like, what do I need to do with it?

Upvotes: 37

Views: 58272

Answers (9)

Gökhan Ayhan
Gökhan Ayhan

Reputation: 1299

That code snippet may help

let nextToken;

do {
  const input = {
    Filters: []
  };
  if (nextToken) {
    input.NextToken = nextToken
  }

  let response = await secretsManagerClient.send(new ListSecretsCommand(input));

  secrets.push(...response.SecretList)
  nextToken = response.NextToken
} while (nextToken)

Upvotes: 0

mario42004
mario42004

Reputation: 11

resp = dynamodb.execute_statement(Statement="SELECT gateway FROM 
Table_Name")
data = resp['Items']
#print(resp['NextToken'])
while 'NextToken' in resp:
    resp = dynamodb.execute_statement(Statement="SELECT gateway FROM 
    Table_Name", NextToken =resp['NextToken'])
    data.extend(resp['Items'])

Upvotes: 1

jono2010
jono2010

Reputation: 719

And another example to return a list of LogGroupNames using nextToken:

    response = logs_client.describe_log_groups(limit=50)
    results = response["logGroups"]
    log.info(f'    response: {response}')
    log.info(f'    results: {results}')
    while "nextToken" in response:
        log.info('    nextToken')
        response = logs_client.describe_log_groups(nextToken=response["nextToken"], limit=50)
        results.extend(response["logGroups"])

    log_group_name_list = []
    for groups in results:
        log_group_name = groups['logGroupName']
        log_group_name_list.append(log_group_name)
    log.info(f'    log_group_name_list: {log_group_name_list}')
    log.info(f'    log_group_name_list:type: {type(log_group_name_list)}')

Upvotes: 0

Rohit kasyap
Rohit kasyap

Reputation: 61

I tried to list secrets names in my secrets manager using boto3 python:

secrets = secret_client.list_secrets()
secrets_manager = (secrets['SecretList'])

for secret in secrets_manager: 
    print ("{0}".format(secret['Name']))

The complete list was around 20, but the output was only 5 secret names.

Updated the code to below, it worked:

secrets = secret_client.list_secrets()
secrets_manager = (secrets['SecretList'])

while "NextToken" in secrets:
    secrets = secret_client.list_secrets(NextToken=secrets["NextToken"])
    secrets_manager.extend(secrets['SecretList'])

for secret in secrets_manager: 
    print ("{0}".format(secret['Name']))

Upvotes: 2

Vincent J
Vincent J

Reputation: 5808

Same as other answer, but with a short snippet with a simple while loop.

response = client.list_accounts()
results = response["Accounts"]
while "NextToken" in response:
    response = client.list_accounts(NextToken=response["NextToken"])
    results.extend(response["Accounts"])

Upvotes: 36

Patrick Ward
Patrick Ward

Reputation: 422

I know it's a different client but for anyone still having issues this took me some time to figure out:

This client required the initial request params also be included. Here's the code:

import boto3

lex_client = boto3.client("lex-models")
response = lex_client.get_slot_types(
    nameContains='search_string',
    maxResults=50
)
slot_types = response['slotTypes']
while 'nextToken' in response.keys():
    response = lex_client.get_slot_types(nameContains='search_string', maxResults=50, nextToken=response['nextToken'])
    slot_types.extend(response['slotTypes'])
print('here are the slot types', slot_types)

Upvotes: 2

Kalim
Kalim

Reputation: 319

Instead you can use get_paginator api. find below example, In my use case i had to get all the values of SSM parameter store and wanted to compare it with a string.

import boto3
import sys

LBURL = sys.argv[1].strip()
client = boto3.client('ssm')
p = client.get_paginator('describe_parameters')
paginator = p.paginate().build_full_result()
for page in paginator['Parameters']:
    response = client.get_parameter(Name=page['Name'])
    value = response['Parameter']['Value']
    if LBURL in value:
        print("Name is: " + page['Name'] + " and Value is: " + value)

Upvotes: 10

Giuseppe Borgese
Giuseppe Borgese

Reputation: 127

Here my example where I used NextToken to check if a secret exist inside the SecretManager. There are also some print useful to visualize the first time

def check_if_secret_existv2(username):
    results_for_call=5
    response = client.list_secrets(MaxResults=results_for_call)
    i=0
    while True:
        i=i+1
        if 'NextToken' in response:
            response = client.list_secrets(MaxResults=results_for_call,NextToken=response['NextToken'])
        else:
            response = client.list_secrets(MaxResults=results_for_call)

        for secret in response['SecretList']:
            print(secret['Name'])
            if secret['Name'] == username:
                return True
        print('End cycle '+str(i))

        if 'NextToken' not in response:
            break
    return False

print(check_if_secret_existv2(myusername))

Upvotes: 2

Torsten Engelbrecht
Torsten Engelbrecht

Reputation: 13496

Don't take the boto3 examples literally (they are not actual examples). Here is how this works:

1) The first time you make a call to list_accounts you'll do it without the NextToken, so simply 

getListAccounts = org_client.list_accounts()

2) This will return a JSON response which looks roughly like this (this is what is saved in your getListAccounts variable): 

{
    "Accounts": [<lots of accounts information>], 
    "NextToken": <some token>
}

Note that the NextToken is only returned in case you have more accounts than one list_accounts call can return, usually this is 100 (the boto3 documentation does not state how many by default). If all accounts were returned in one call there is no NextToken in the response!

3) So if and only if not all accounts were returned in the first call you now want to return more accounts and you will have to use the NextToken in order to do this: 

getListAccountsMore = org_client.list_accounts(NextToken=getListAccounts['NextToken'])

4) Repeat until no NextToken is returned in the response anymore (then you retrieved all accounts).

This is how the AWS SDK handles pagination in many cases. You will see the usage of the NextToken in other service clients as well.

Upvotes: 37

Related Questions