CODEWITHSUNDEEP
c#elasticsearchnest
M.frank
M.frank

Reputation: 145

Elasticsearch infer _id on dynamic type when using IndexMany

I am trying to get my head around a problem. I am building an application where we are indexing assets in Elastic. The nature of the assets is very dynamic, because they contain client metadata, which is different from client to client.

Because of this, the Index is built from a List of dynamics in C#. This actually works like a charm. The problem is, I cannot control the _id property in Elastic when using the C# interface. This means when I update the documents, instead of updating the correct one a new duplicate is made.

My code looks like this:

List<dynamic> assets = new List<dynamic>();
var settings1 = new ConnectionSettings(
    new Uri("http://localhost:9200")
    ).DefaultIndex("assets");

var client = new ElasticClient(settings1);

//assets is build here

var indexResponse = client.Indices.Create("assets");
var BulkResponse = client.IndexMany(assets);

This actually works and the index is built as I expect it to - almost. Even though I have a property called Id on the dynamic, it is not inferred correctly, which means the document is given an _Id decided by Elastic. Thus the next time I run this code using the same Id a new document is created rather than updated.

I have been searching high and low, but cannot seem to find a good solution. One thing I have tried is the following:

var bulkResponse = client.Bulk(bd => bd.IndexMany(assets, (descriptor, s) => descriptor.Id(s.Id)));

But this throws an error I cannot catch in the .net kernel. This actually works on lower versions on Elastic, but seems to have been broken with 7.2 and 7.0.1 of the C# interface.

Any help is much appreciated.

Upvotes: 1

Views: 824

Answers (2)

Russ Cam
Russ Cam

Reputation: 125518

To allow the following to work

var bulkResponse = client.Bulk(bd => bd.IndexMany(assets, (descriptor, s) => descriptor.Id(s.Id)));

You just need to cast the Id type to the type that it is. For example, if it's a string

var client = new ElasticClient();

var assets = new dynamic[] 
{
    new { Id = "1", Name = "foo" },
    new { Id = "2", Name = "bar" },
    new { Id = "3", Name = "baz" },     
};

var bulkResponse = client.Bulk(bd => bd.IndexMany(assets, (descriptor, s) => descriptor.Id((string)s.Id)));

This is a runtime limitation.

Upvotes: 3

Rob
Rob

Reputation: 9979

Instead of using dynamic type you could create dictionary-based custom type like:

    public class DynamicDocument : Dictionary<string, object>
    {
        public string Id => this["id"]?.ToString();
    }

and use it as follow:

class Program
{
    public class DynamicDocument : Dictionary<string, object>
    {
        public string Id => this["id"]?.ToString();
    }

    static async Task Main(string[] args)
    {
        var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
        var connectionSettings = new ConnectionSettings(pool);
        connectionSettings.DefaultIndex("documents");

        var client = new ElasticClient(connectionSettings);

        await client.Indices.DeleteAsync("documents");
        await client.Indices.CreateAsync("documents");

        var response = await client.IndexAsync(
            new DynamicDocument
            {
                {"id", "1"}, 
                {"field1", "value"}, 
                {"field2", 1}
            }, descriptor => descriptor);

        //will update document with id 1 as it's already exists
        await client.IndexManyAsync(new[]
        {
            new DynamicDocument
            {
                {"id", "1"},
                {"field1", "value2"},
                {"field2", 2}
            }
        }); 

        await client.Indices.RefreshAsync();

        var found = await client.GetAsync<DynamicDocument>("1");

        Console.WriteLine($"Id: {found.Source.Id}");
        Console.WriteLine($"field1: {found.Source["field1"]}");
        Console.WriteLine($"field2: {found.Source["field2"]}");
    }
}

Output:

Id: 1
field1: value2
field2: 2

Tested with elasticsearch 7.2.0 and NEST 7.0.1.

Hope that helps.

Upvotes: 2

Related Questions