CODEWITHSUNDEEP
amazon-web-servicesweb-scrapingboto3botocoreamazon-marketplace
BiggestPotato
BiggestPotato

Reputation: 19

List all available Data Products and their metadata available in AWS Marketplace

AWS Marketplace has a number of different datasets available in their 'Data Products' category

https://aws.amazon.com/marketplace/b/d5a43d97-558f-4be7-8543-cce265fe6d9d?ref_=mp_nav_category_d5a43d97-558f-4be7-8543-cce265fe6d9d&category=d5a43d97-558f-4be7-8543-cce265fe6d9d

I'd like to find a way to programatically parse this catalogue and extract some metadata for each of these data products, such as their descriptions, so I can aggregate it in a file.

I've tried using the code snippets below, but they seem to only provide what the account is currently subscribed to, not all of whats available

client = boto3.client('marketplace-catalog',
                      region_name='us-east-1')

response = client.list_entities(
    Catalog='AWSMarketplace',
    EntityType='DataProduct'
)

return response['EntitySummaryList']

and

client = boto3.client('dataexchange',
                       region_name='us-east-1'
)

response = client.list_data_sets()

return response['DataSets']

In both cases, EntitySummaryList and DataSets are empty.

Is there any way to do this without webscraping? I'm also somewhat familiar with the hack of using 'describe-images' with AWS marketplace as the owner, but it doesn't seem to be a neat solution

Upvotes: 1

Views: 40

Answers (0)

Related Questions