CODEWITHSUNDEEP
c#azure-blob-storage
Wiebe Tijsma
Wiebe Tijsma

Reputation: 10265

Connecting to azurite using a hostname fails

UPDATE: I can verify this behavior is fixed in Azure.Storage.Blobs 12.5.1 https://www.nuget.org/packages/Azure.Storage.Blobs https://github.com/Azure/azure-sdk-for-net/issues/9404

How can I connect to azurite using a hostname?

I'm trying to emulate Azure Blob Storage in docker using Azurite for integration tests.

All works well, to the point I have to access Azurite via a hostname (which is AFAIK required for docker networking)

My connection string looks like this (which is the default well-known connection string):

"AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;DefaultEndpointsProtocol=http;BlobEndpoint=http://azurite:10000/devstoreaccount1;"

My docker compose part for azurite looks like this:

services:
  azurite:
    image: mcr.microsoft.com/azure-storage/azurite
    hostname: azurite
    command: "azurite-blob --loose --blobHost 0.0.0.0"
    ports:
      - "10000:10000"
    volumes:
      - ./test/azurite:/data
    networks:
      - stillsnet

  images:
    container_name: images
    image: myapp/images
    build:
      context: .
      dockerfile: Dockerfile
    ports:
       - "5000:5000"
       - "5001:5001"
    environment:
      - ASPNETCORE_ENVIRONMENT=Test
      - ASPNETCORE_URLS=http://+:5000
      - imagesStorage__AzureBlobStorage__ConnectionString=AccountName=devstoreaccount1;DefaultEndpointsProtocol=http;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://azurite:10000;
    depends_on:
      - azurite
    links:
      - azurite
    networks:
      - stilssnet

my code looks like this:

private const string ConnectionString ="AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;DefaultEndpointsProtocol=http;BlobEndpoint=http://azurite:10000/devstoreaccount1;";

[Fact]
public async Task UploadFile()
{
   var container = new BlobContainerClient(ConnectionString, "images");
   await using var stream = File.OpenRead(@"C:\temp\output\3ee9bc41-40ea-4d05-b180-e74bd5065622\images\00000000.jpg");
   await container.UploadBlobAsync("test.jpg", stream);
}

this will throw an exception:

System.Xml.XmlException : Root element is missing.
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
   at System.Xml.XmlTextReaderImpl.Read()
   at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)
   at System.Xml.Linq.XDocument.Load(Stream stream, LoadOptions options)
   at Azure.Storage.Blobs.BlobRestClient.Container.CreateAsync_CreateResponse(Response response)
   at Azure.Storage.Blobs.BlobRestClient.Container.CreateAsync(ClientDiagnostics clientDiagnostics, HttpPipeline pipeline, Uri resourceUri, PublicAccessType access, Nullable`1 timeout, IDictionary`2 metadata, String requestId, Boolean async, String operationName, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.BlobContainerClient.CreateInternal(PublicAccessType publicAccessType, IDictionary`2 metadata, Boolean async, CancellationToken cancellationToken, String operationName)
   at Azure.Storage.Blobs.BlobContainerClient.CreateIfNotExistsInternal(PublicAccessType publicAccessType, IDictionary`2 metadata, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.BlobContainerClient.CreateIfNotExistsAsync(PublicAccessType publicAccessType, IDictionary`2 metadata, CancellationToken cancellationToken)

If I change the connection string from azurite to 127.0.0.1 it all works fine.

Upvotes: 11

Views: 15446

Answers (6)

Adamy
Adamy

Reputation: 2849

I discovered an issue with the Azurite container where the host name must be a single word, such as 'azurite' or 'myazurite.' If you use a container/host name that includes a period, like 'azurite.mydomain' or 'test.mystorage,' it will fail, and the Azure client will return a '404 Resource not found' error.

Upvotes: 0

Wiebe Tijsma
Wiebe Tijsma

Reputation: 10265

UPDATE: I can verify this behavior is fixed in Azure.Storage.Blobs 12.5.1 https://www.nuget.org/packages/Azure.Storage.Blobs https://github.com/Azure/azure-sdk-for-net/issues/9404

It looks like the BlobContainerClient itself strips away the account name from the URL if you use anything else than localhost or an IP address, so the client generates this:

PUT /images?restype=container HTTP/1.1" 400 - instead of PUT /devstoreaccount1/images?restype=container HTTP/1.1 201

As an ugly workaround we can include the account name in the container name when testing against azurite: var container = new BlobContainerClient(ConnectionString, "devstoreaccount1/images");

Though await container.CreateIfNotExistsAsync(); doesn't work properly against azurite then (throws a 409 Exception when it already exist...

So we either have:

  • an ugly hack
  • dropping azurite in favor of a real blob storage account
  • dropping the BlobContainerClient which seems to perform too much magic based on the host name.

It doesn't seem to be related to Azurite itself, or ideally it should support root URL's that are compatible with the Azure blob store without having to specify the account name prefix.

Upvotes: 3

wwww
wwww

Reputation: 822

Root of the problem is how internals of azure-storage-queue library split URL. My problem was fix by usage of newer version of connector. I know that it's c# thread but in java I had the same problem. Please use 12.6.0-beta.1 or higher if version number is the same in c#.

<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-storage-queue</artifactId>
    <version>12.6.0-beta.1</version>
</dependency>

version 12.0.1 doesn't seem to work properly.

Upvotes: 1

Isaac Ojeda
Isaac Ojeda

Reputation: 325

The host's name in that network is azurite, so you can use this connection string UseDevelopmentStorage=true;DevelopmentStorageProxyUri=http://azurite.

Works for me and I think is the way to do it.

Also you can use the legacy storage emulator (that includes Tables), instead of azurite you put docker.host.internal

Upvotes: 7

Mike
Mike

Reputation: 607

I run both Sql server and Azurite. I just use docker compose to start them up. Notice the restart: always. this will start the containers when docker starts . It's just like running the installed azure storage emulator.

version: "2"

services:

  sql-server-db:
    container_name: sql-server-db
    image: microsoft/mssql-server-linux:2017-latest
    restart: always
    ports:
      - "1433:1433"
      - "1434:1434"
    environment:
      SA_PASSWORD: "<EnterYourPasswordHere!!!>"
      ACCEPT_EULA: "Y"

  azure_storage_emulator:
    container_name: azurite
    image: mcr.microsoft.com/azure-storage/azurite
    restart: always
    ports:
      - "10001:10001"
      - "10000:10000"

Upvotes: 0

user9705827
user9705827

Reputation: 1

I ran the azurite with this command (from the docker hub page https://hub.docker.com/_/microsoft-azure-storage-azurite?tab=description)

docker run -p 10000:10000 -p 10001:10001 mcr.microsoft.com/azure-storage/azurite

my config file is just this:

<add name="BlobStorage" connectionString="UseDevelopmentStorage=true" />

in the connectionStrings section (from the page: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite#install-and-run-the-azurite-docker-image)

Upvotes: 0

Related Questions