Azure.QueueClient: error when connecting to host.docker.internal

Question

I am experimenting with Aspire. When you create an Azure Storage in Aspire and try to inject the connection string to another container it resolves with hostname: host.docker.internal.

var storage = builder.AddAzureStorage("Storage");

if (builder.Environment.IsDevelopment())
{
    storage.RunAsEmulator(
        azuriteBuilder =>
        {
            azuriteBuilder
                .WithArgs(
                [
                    "docker-entrypoint.sh",
                    "azurite",
                    "--loose",
                    "-l",
                    "/data",
                    "--blobHost",
                    "0.0.0.0",
                    "--queueHost",
                    "0.0.0.0",
                    "--tableHost",
                    "0.0.0.0",
                    "--disableProductStyleUrl",
                    "--debug",
                    "/data/debug.log"
                ])
                .WithQueuePort(53801)
                .WithBlobPort(53800);
        });
}

var blobs = storage.AddBlobs("blobs");
var queues = storage.AddQueues("queues");

builder.AddContainer("kernel-memory", "kernelmemory/service", "latest")
    .WithImageRegistry("docker.io")
    .WithBindMount("kernel.memory.appsettings.Development.json", "/app/appsettings.json")
    .WithEnvironment("KernelMemory__Services__AzureBlobs__ConnectionString", blobs)
    .WithEnvironment("KernelMemory__Services__AzureQueues__ConnectionString", queues)
    ;

Example connection string:

DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;QueueEndpoint=http://host.docker.internal:53801/devstoreaccount1;

As I read this is a way of communicating between two containers on the same host machine.

In my consumer container I have a .NET application running Azure.Storage.Queues -v 12.19.1 nuget package.

When I tried to create an Queue I got an error from Azurite:

2024-09-07T10:02:33.031Z 1250b51a-ba5f-429d-8535-48e57436f29f info: QueueStorageContextMiddleware: RequestMethod=PUT RequestURL=http://host.docker.internal/delete-generated-files RequestHeaders:{"host":"host.docker.internal:53801","x-ms-version":"2018-11-09","accept":"application/xml","x-ms-client-request-id":"142f3ebd-e15a-41fd-95fb-176cab693f8c","x-ms-return-client-request-id":"true","user-agent":"azsdk-net-Storage.Queues/12.18.0 (.NET 8.0.7; Alpine Linux v3.20)","x-ms-date":"Sat, 07 Sep 2024 10:02:27 GMT","authorization":"SharedKey devstoreaccount1:w+IZfvW8qAa3Bv6Tu93HD5fB4DVEolR8A5Dv0w9mCK0=","content-length":"0"} ClientIP=192.168.127.1 Protocol=http HTTPVersion=1.1
2024-09-07T10:02:33.035Z 1250b51a-ba5f-429d-8535-48e57436f29f info: QueueStorageContextMiddleware: Account=delete-generated-files Queue=undefined Message=undefined MessageId=undefined
2024-09-07T10:02:33.037Z 1250b51a-ba5f-429d-8535-48e57436f29f verbose: DispatchMiddleware: Dispatching request...
2024-09-07T10:02:33.041Z 1250b51a-ba5f-429d-8535-48e57436f29f error: DispatchMiddleware: Incoming URL doesn't match any of swagger defined request patterns.
2024-09-07T10:02:33.042Z 1250b51a-ba5f-429d-8535-48e57436f29f error: ErrorMiddleware: Received a MiddlewareError, fill error information to HTTP response
2024-09-07T10:02:33.043Z 1250b51a-ba5f-429d-8535-48e57436f29f error: ErrorMiddleware: ErrorName=UnsupportedRequestError ErrorMessage=Incoming URL doesn't match any of swagger defined request patterns.  ErrorHTTPStatusCode=400 ErrorHTTPStatusMessage=undefined ErrorHTTPHeaders=undefined ErrorHTTPBody=undefined ErrorStack="UnsupportedRequestError: Incoming URL doesn't match any of swagger defined request patterns.\n    at dispatchMiddleware (/opt/azurite/dist/src/queue/generated/middleware/dispatch.middleware.js:41:30)\n    at /opt/azurite/dist/src/queue/generated/ExpressMiddlewareFactory.js:49:47\n    at Layer.handle [as handle_request] (/opt/azurite/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/opt/azurite/node_modules/express/lib/router/index.js:328:13)\n    at /opt/azurite/node_modules/express/lib/router/index.js:286:9\n    at Function.process_params (/opt/azurite/node_modules/express/lib/router/index.js:346:12)\n    at next (/opt/azurite/node_modules/express/lib/router/index.js:280:10)\n    at queueStorageContextMiddleware (/opt/azurite/dist/src/queue/middlewares/queueStorageContext.middleware.js:97:5)\n    at /opt/azurite/dist/src/queue/middlewares/queueStorageContext.middleware.js:15:16\n    at Layer.handle [as handle_request] (/opt/azurite/node_modules/express/lib/router/layer.js:95:5)"
2024-09-07T10:02:33.043Z 1250b51a-ba5f-429d-8535-48e57436f29f error: ErrorMiddleware: Set HTTP code: 400
2024-09-07T10:02:33.043Z 1250b51a-ba5f-429d-8535-48e57436f29f error: ErrorMiddleware: Set HTTP body: undefined

In my debugging I figured out this must be a problem in the QueueClient used in my .NET Application.

After debugging I noticed that the QueueClient does not register the AccountName:

I believe that the problem must be in QueueClient messing up the connection string registration. I tried by adding UseDevelopmentStorage=true to the connection string but it did not work.

Being stuck on this problem for quite some time. I would appreciate any help or suggestions.

Answer 1

I tried your code and encountered the same error . The issue is with connection string contained host.docker.internal. To allow one container to connect to another container using the host’s connection details with host.docker.internal, you need add the connection string directly into the environment variables of the container that needs to connect.

"DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;QueueEndpoint=http://${DOCKER_GATEWAY_HOST:-host.docker.internal}:53801/devstoreaccount1;"

Thank @Stefan Natter for explanation.I have refered this blog to host’s IP Address inside a Docker container.

In general, the Azurite emulator for local Azure Storage development runs on port 10001. To run it on port 53801, you can use a docker-compose.yml file in C# as shown below, or use the docker run command.

I referred to this link to pass environment variables to Docker containers.

docker-compose.yml:

version: '3.8'
services:
  azurite:
    image: mcr.microsoft.com/azure-storage/azurite
    ports:
      - "53800:10000"
      - "53801:10001" 
      - "53802:10002" 

  csharp-app:
    build: .
    depends_on:
      - azurite
    environment:
      - AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;QueueEndpoint=http://${DOCKER_GATEWAY_HOST:-host.docker.internal}:53801/devstoreaccount1;"
     depends_on:
      - other_service`
    env_file:
      - .env

using docker run cmd to run under 53801 port.

docker run  --name azurite -p 53800:10000 -p 53801:10001 -p 53802:10002  mcr.microsoft.com/azure-storage/azurite azurite --blobHost 0.0.0.0 --blobPort 10000 --queueHost 0.0.0.0 --queuePort 10001 --tableHost 0.0.0.0 --tablePort 10002 --loose --debug /data/debug.log

The code below is used to create a queue in the Azure Storage Emulator. The code is taken from GitHub. Thank you, @radical.

 string connectionString = Environment.GetEnvironmentVariable("STORAGE_CONNECTION_STRING");
        string queueName = "quickstartqueues-" + Guid.NewGuid().ToString();
        QueueClient queueClient = new QueueClient(connectionString, queueName);
        Console.WriteLine($"Creating queue: {queueName}");
        await queueClient.CreateAsync();