How to connect to the CosmosDB Emulator's table storage using python?

Question

I'm running the Azure CosmosDB Emulator locally, and I wish to use the Table interface from Python. Microsoft provides the azure-cosmosdb-table python sdk for connecting to the Table API [1]. I cannot get connections working successfully to the emulator, even though I am able to connect to an actual CosmosDB account in the cloud.

Version numbers etc:

• Windows 10
• Python 3.6 (CPython)
• azure-cosmosdb-table 1.0.5

The code I'm using for trying to access the emulator:

import logging
from azure.cosmosdb.table import TableService

from nossl import no_ssl_verification

logging.basicConfig(level=logging.DEBUG)

connection = {
    'AccountName': 'devstoreaccount1',
    # 'AccountEndpoint': 'https://localhost:8081/',
    'AccountKey': 'C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw=="',
    # 'EndpointSuffix': 'localhost:8081',
    # 'DefaultEndpointsProtocol': 'https',
    'TableEndpoint': 'https://localhost:8081',
}
connection_string = ';'.join('{k}={v}'.format(k=k, v=v) for k, v in connection.items())
print(connection_string)

with no_ssl_verification():
    ts = TableService(
        endpoint_suffix = "https://localhost:8081",
        connection_string= connection_string,
        is_emulated=True)
    ts.create_table('mytable')

The no_ssl_verification context manager disables all SSL verification in the requests library, which is used by the SDK. It was copied from an answer here on Stack Overflow [5].

The example code above produces the following error:

AccountName=devstoreaccount1;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==";TableEndpoint=https://localhost:8081
DEBUG:azure.cosmosdb.table._auth:String_to_sign=POST

application/json
Thu, 18 Apr 2019 13:50:42 GMT
/devstoreaccount1/Tables
INFO:azure.storage.common.storageclient:Client-Request-ID=f52a8b36-61e0-11e9-8fc0-a44cc8cee022 Outgoing request: Method=POST, Path=/Tables, Query={'timeout': None}, Headers={'Content-Type': 'application/json', 'Prefer': 'return-no-content', 'Accept': 'application/json;odata=minimalmetadata', 'DataServiceVersion': '3.0;NetFx', 'MaxDataServiceVersion': '3.0', 'Content-Length': '24', 'x-ms-version': '2018-03-28', 'User-Agent': 'Azure-Storage/1.4.0-None (Python CPython 3.6.4;
Windows 10)', 'x-ms-client-request-id': 'f52a8b36-61e0-11e9-8fc0-a44cc8cee022', 'x-ms-date': 'Thu, 18 Apr 2019 13:50:42 GMT', 'Authorization': 'REDACTED'}.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): localhost:8081
DEBUG:urllib3.connectionpool:https://localhost:8081 "POST /Tables HTTP/1.1" 400 None
INFO:azure.storage.common.storageclient:Client-Request-ID=f52a8b36-61e0-11e9-8fc0-a44cc8cee022 Receiving Response: Server-Timestamp=Thu, 18 Apr 2019 13:50:42 GMT, HTTP Status Code=400, Message=Bad Request, Headers={'transfer-encoding': 'chunked', 'content-type': 'application/json', 'server': 'Microsoft-HTTPAPI/2.0',
'x-ms-activity-id': '00000000-0000-0000-0000-000000000000', 'x-ms-gatewayversion': 'version=2.2.0.0', 'date': 'Thu, 18 Apr 2019 13:50:42 GMT'}.
INFO:azure.storage.common.storageclient:Client-Request-ID=f52a8b36-61e0-11e9-8fc0-a44cc8cee022 Operation failed: checking if the operation should be retried. Current retry count=0, Server-Timestamp=Thu, 18 Apr 2019 13:50:42 GMT, HTTP status code=400, Exception=Bad Request{"code":"BadRequest","message":"Request url is invalid., Microsoft.Azure.Documents.Common/2.2.0.0"}.
ERROR:azure.storage.common.storageclient:Client-Request-ID=f52a8b36-61e0-11e9-8fc0-a44cc8cee022 Retry policy did not allow for a retry: Server-Timestamp=Thu, 18 Apr 2019 13:50:42 GMT, HTTP status code=400, Exception=Bad Request{"code":"BadRequest","message":"Request url is invalid., Microsoft.Azure.Documents.Common/2.2.0.0"}.
Traceback (most recent call last):
  File "c:/Users/josteb/sandbox/cosmoskvstore/examples.py", line 24, in <module>
    ts.create_table('mytable')
  File "C:\Users\josteb\.virtualenvs\cosmoskvstore-cPSP36p_\lib\site-packages\azure\cosmosdb\table\tableservice.py", line 539, in create_table
    _dont_fail_on_exist(ex)
  File "C:\Users\josteb\.virtualenvs\cosmoskvstore-cPSP36p_\lib\site-packages\azure\storage\common\_error.py", line 88, in _dont_fail_on_exist
    raise error
  File "C:\Users\josteb\.virtualenvs\cosmoskvstore-cPSP36p_\lib\site-packages\azure\cosmosdb\table\tableservice.py", line 536, in create_table
    self._perform_request(request)
  File "C:\Users\josteb\.virtualenvs\cosmoskvstore-cPSP36p_\lib\site-packages\azure\cosmosdb\table\tableservice.py", line 1106, in _perform_request
    return super(TableService, self)._perform_request(request, parser, parser_args, operation_context)
  File "C:\Users\josteb\.virtualenvs\cosmoskvstore-cPSP36p_\lib\site-packages\azure\storage\common\storageclient.py", line 430, in _perform_request
    raise ex
  File "C:\Users\josteb\.virtualenvs\cosmoskvstore-cPSP36p_\lib\site-packages\azure\storage\common\storageclient.py", line 358, in _perform_request
    raise ex
  File "C:\Users\josteb\.virtualenvs\cosmoskvstore-cPSP36p_\lib\site-packages\azure\storage\common\storageclient.py", line 344, in _perform_request
    HTTPError(response.status, response.message, response.headers, response.body))
  File "C:\Users\josteb\.virtualenvs\cosmoskvstore-cPSP36p_\lib\site-packages\azure\storage\common\_error.py", line 115, in _http_error_handler
    raise ex
azure.common.AzureHttpError: Bad Request
{"code":"BadRequest","message":"Request url is invalid., Microsoft.Azure.Documents.Common/2.2.0.0"}

I have not been able to find the connection parameters to be used for the emulator properly documented anywhere. The web interface of the emulator provides the account name and the account key. The rest is a bit of guesswork, which I arrived at through experimentation. Without these extra parameters, I can see from the log output that the SDK is attempting to contact a remote host (likely part of the cloud-based Cosmos service), which of course fails. The same goes for the is_emulated flag; when False, connections are made to a remote host.

To see if there was something wrong with the emulator, I tried to use the azure-cosmos SDK for the SQL/document interface of cosmosdb to access the emulator. That worked just fine with the following code:

from azure.cosmos.cosmos_client import CosmosClient

cc = CosmosClient('https://localhost:8081', auth={'masterKey': 'C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw=='})
cc.CreateDatabase({'id': 'mydb'})

This completed without problems, and I can see in the Emulator's data explorer that the database has been created.

Does anyone have a reference for the correct connection string for the table interface of the CosmosDB emulator?

[1] https://learn.microsoft.com/en-us/python/api/azure-cosmosdb-table/?view=azure-python

[2] https://learn.microsoft.com/en-us/python/api/azure-cosmos/?view=azure-python

[3] https://learn.microsoft.com/en-us/azure/cosmos-db/local-emulator-export-ssl-certificates

[4] https://bugs.python.org/issue28547

[5] How do I disable the security certificate check in Python requests

Answer 1

A way to disable certificate validation in the python SDK for table storage

After deep debugging the create_table function,I found you could the verify property in the session.py in the requests package.

The default value is True,you could set it as False to skip the SSL Verification.

A way to get the certificates validated

Another property named cert which could configure as cert path.

In addition,please refer to suggestions in this document and this ssl configuration link.