Reputation: 2785
Elastic NEST Client how to sort by on a nested property
As a newbie to NEST Library, pulling my hair out with the documentation that's making no sense to me at all.
First Issue: Cannot figure out how to map the results from a query to a strongly type model, despite having captured the json result and pasting to a C# class, what ever I try, unless I use a dynamic type in the query method then the results return null.
Model derived from the json result returned from the query:
public class Rootobject
{
public int took { get; set; }
public bool timed_out { get; set; }
public _Shards _shards { get; set; }
public Hits hits { get; set; }
}
public class _Shards
{
public int total { get; set; }
public int successful { get; set; }
public int skipped { get; set; }
public int failed { get; set; }
}
public class Hits
{
public Total total { get; set; }
public float max_score { get; set; }
public Hit[] hits { get; set; }
}
public class Total
{
public int value { get; set; }
public string relation { get; set; }
}
public class Hit
{
public string _index { get; set; }
public string _type { get; set; }
public string _id { get; set; }
public float _score { get; set; }
public _Source _source { get; set; }
}
public class _Source
{
public DateTime timestamp { get; set; }
public string level { get; set; }
public string messageTemplate { get; set; }
public string message { get; set; }
public Fields fields { get; set; }
}
public class Fields
{
public string LogEventCategory { get; set; }
public string LogEventType { get; set; }
public string LogEventSource { get; set; }
public string LogData { get; set; }
public string MachineName { get; set; }
public int MemoryUsage { get; set; }
public int ProcessId { get; set; }
public string ProcessName { get; set; }
public int ThreadId { get; set; }
}
Sample JSON result returned from query:
{
"took" : 16,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 8,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "webapp-razor-2021.05",
"_type" : "_doc",
"_id" : "n2tbTnkBwE4YgJowzRsT",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2021-05-09T00:41:47.2321845+01:00",
"level" : "Information",
"messageTemplate" : "{@LogEventCategory}{@LogEventType}{@LogEventSource}{@LogData}",
"message" : "\"WebApp-RAZOR\"\"Application Startup\"\"System\"\"Application Starting Up\"",
"fields" : {
"LogEventCategory" : "WebApp-RAZOR",
"LogEventType" : "Application Startup",
"LogEventSource" : "System",
"LogData" : "Application Starting Up",
"MachineName" : "DESKTOP-OS52032",
"MemoryUsage" : 4713408,
"ProcessId" : 15152,
"ProcessName" : "WebApp-RAZOR",
"ThreadId" : 1
}
}
},
{
"_index" : "webapp-razor-2021.05",
"_type" : "_doc",
"_id" : "oGtdTnkBwE4YgJowuxu_",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2021-05-09T00:43:54.0326968+01:00",
"level" : "Information",
"messageTemplate" : "{@LogEventCategory}{@LogEventType}{@LogEventSource}{@LogData}",
"message" : "\"WebApp-RAZOR\"\"Application Startup\"\"System\"\"Application Starting Up\"",
"fields" : {
"LogEventCategory" : "WebApp-RAZOR",
"LogEventType" : "Application Startup",
"LogEventSource" : "System",
"LogData" : "Application Starting Up",
"MachineName" : "DESKTOP-OS52032",
"MemoryUsage" : 4656048,
"ProcessId" : 12504,
"ProcessName" : "WebApp-RAZOR",
"ThreadId" : 1
}
}
},
{
"_index" : "webapp-razor-2021.05",
"_type" : "_doc",
"_id" : "oWtgTnkBwE4YgJownRtc",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2021-05-09T00:47:02.8954368+01:00",
"level" : "Information",
"messageTemplate" : "{@LogEventCategory}{@LogEventType}{@LogEventSource}{@LogData}",
"message" : "\"WebApp-RAZOR\"\"Application Startup\"\"System\"\"Application Starting Up\"",
"fields" : {
"LogEventCategory" : "WebApp-RAZOR",
"LogEventType" : "Application Startup",
"LogEventSource" : "System",
"LogData" : "Application Starting Up",
"MachineName" : "DESKTOP-OS52032",
"MemoryUsage" : 4717560,
"ProcessId" : 17952,
"ProcessName" : "WebApp-RAZOR",
"ThreadId" : 1
}
}
}
]
}
}
2nd Issue: I'm trying to sort by a property that is nested within the JSON, an example being the property called "LogEventCategory"
NEST Client method:
var searchResponse = await _elasticClient.SearchAsync<dynamic>(s => s
.RequestConfiguration(r => r
.DisableDirectStreaming()
)
//.AllIndices()
.From(0) // From parameter defines the offset from the first result you want to fetch.
.Size(3) // Size parameter allows you to configure the maximum amount of hits to be returned.
.Index("webapp-razor-*")
//.Index("index-1,index-2")
.Query(q => q
.MatchAll()
)
.Sort(so => so
.Field(fs => fs
//.Field("@timestamp") // this one seems to work
.Field("logEventCategory")
.Order(SortOrder.Ascending)
//.Order(ColumnSortOrder)
)
)
);
Knowing that NEST High Level Client is designed to map against proper models, I realise using dynamic type is not correct, but I just can't figure our why using a class that was created/mapped directly from a sample json response is still not working and that the result returned when doing so just gives me null values for the first few properties.
Example below where only the first few items are seen (but still null)
Upvotes: 0
Views: 2451
Answers (1)
Reputation: 125528
First Issue: Cannot figure out how to map the results from a query to a strongly type model, despite having captured the json result and pasting to a C# class, what ever I try, unless I use a dynamic type in the query method then the results return null.
Model derived from the json result returned from the query:
You don't need to define models for the entire JSON response; the client will take care of correctly deserializing most of the response, so the only POCO that needs to be created is for the _source JSON object. Based on the example provided, this would be something like
public class LogMessage
{
[PropertyName("@timestamp")]
public DateTimeOffset Timestamp { get; set; }
public string Level {get;set;}
public string MessageTemplate {get;set;}
public string Message {get;set;}
public Dictionary<string, object> Fields {get;set;}
}
We can check to see if this deserializes correctly by creating a stub response and passing it to the client to deserialize
private static void Main()
{
var defaultIndex = "default_index";
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var json = @"{
""took"" : 16,
""timed_out"" : false,
""_shards"" : {
""total"" : 1,
""successful"" : 1,
""skipped"" : 0,
""failed"" : 0
},
""hits"" : {
""total"" : {
""value"" : 8,
""relation"" : ""eq""
},
""max_score"" : 1.0,
""hits"" : [
{
""_index"" : ""webapp-razor-2021.05"",
""_type"" : ""_doc"",
""_id"" : ""n2tbTnkBwE4YgJowzRsT"",
""_score"" : 1.0,
""_source"" : {
""@timestamp"" : ""2021-05-09T00:41:47.2321845+01:00"",
""level"" : ""Information"",
""messageTemplate"" : ""{@LogEventCategory}{@LogEventType}{@LogEventSource}{@LogData}"",
""message"" : ""\""WebApp-RAZOR\""\""Application Startup\""\""System\""\""Application Starting Up\"""",
""fields"" : {
""LogEventCategory"" : ""WebApp-RAZOR"",
""LogEventType"" : ""Application Startup"",
""LogEventSource"" : ""System"",
""LogData"" : ""Application Starting Up"",
""MachineName"" : ""DESKTOP-OS52032"",
""MemoryUsage"" : 4713408,
""ProcessId"" : 15152,
""ProcessName"" : ""WebApp-RAZOR"",
""ThreadId"" : 1
}
}
},
{
""_index"" : ""webapp-razor-2021.05"",
""_type"" : ""_doc"",
""_id"" : ""oGtdTnkBwE4YgJowuxu_"",
""_score"" : 1.0,
""_source"" : {
""@timestamp"" : ""2021-05-09T00:43:54.0326968+01:00"",
""level"" : ""Information"",
""messageTemplate"" : ""{@LogEventCategory}{@LogEventType}{@LogEventSource}{@LogData}"",
""message"" : ""\""WebApp-RAZOR\""\""Application Startup\""\""System\""\""Application Starting Up\"""",
""fields"" : {
""LogEventCategory"" : ""WebApp-RAZOR"",
""LogEventType"" : ""Application Startup"",
""LogEventSource"" : ""System"",
""LogData"" : ""Application Starting Up"",
""MachineName"" : ""DESKTOP-OS52032"",
""MemoryUsage"" : 4656048,
""ProcessId"" : 12504,
""ProcessName"" : ""WebApp-RAZOR"",
""ThreadId"" : 1
}
}
},
{
""_index"" : ""webapp-razor-2021.05"",
""_type"" : ""_doc"",
""_id"" : ""oWtgTnkBwE4YgJownRtc"",
""_score"" : 1.0,
""_source"" : {
""@timestamp"" : ""2021-05-09T00:47:02.8954368+01:00"",
""level"" : ""Information"",
""messageTemplate"" : ""{@LogEventCategory}{@LogEventType}{@LogEventSource}{@LogData}"",
""message"" : ""\""WebApp-RAZOR\""\""Application Startup\""\""System\""\""Application Starting Up\"""",
""fields"" : {
""LogEventCategory"" : ""WebApp-RAZOR"",
""LogEventType"" : ""Application Startup"",
""LogEventSource"" : ""System"",
""LogData"" : ""Application Starting Up"",
""MachineName"" : ""DESKTOP-OS52032"",
""MemoryUsage"" : 4717560,
""ProcessId"" : 17952,
""ProcessName"" : ""WebApp-RAZOR"",
""ThreadId"" : 1
}
}
}
]
}
}";
// create a connection that always returns the above response
var connection = new InMemoryConnection(Encoding.UTF8.GetBytes(json));
var settings = new ConnectionSettings(pool, connection)
.DefaultIndex(defaultIndex);
var client = new ElasticClient(settings);
// actually doesn't matter what query we send through as we'll always get the stubbed response
var searchResponse = client.Search<LogMessage>(s => s);
foreach (var document in searchResponse.Documents)
{
Console.WriteLine($"{document.Timestamp} {document.Message}");
}
}
which yields
9/05/2021 12:41:47 AM +01:00 "WebApp-RAZOR""Application Startup""System""Application Starting Up"
9/05/2021 12:43:54 AM +01:00 "WebApp-RAZOR""Application Startup""System""Application Starting Up"
9/05/2021 12:47:02 AM +01:00 "WebApp-RAZOR""Application Startup""System""Application Starting Up"
2nd Issue: I'm trying to sort by a property that is nested within the JSON, an example being the property called "LogEventCategory"
In LogMessage POCO, the fields are mapped as a Dictionary<string,object> as I guess the key/values can be dynamic. To sort on a property in this object would be
var searchResponse = client.Search<LogMessage>(s => s
.Sort(so => so
.Ascending(f => f.Fields["LogEventCategory"])
)
);
which yields the following request
POST http://localhost:9200/default_index/_search?pretty=true&typed_keys=true
{
"sort": [
{
"fields.LogEventCategory": {
"order": "asc"
}
}
]
}
If Elasticsearch has inferred the mapping for LogEventCategory i.e. no explicit data type mapping was provided for it, then it'll be mapped as a text field with a keyword multi-field (or sub-field). Attempting to sort on a text field where fielddata has not been enabled (it's disabled by default) will result in an error. One typically should use a keyword field for sorting and aggregations, so if the mapping has been inferred, the keyword multi-field can be targeted for the sorting with
var searchResponse = client.Search<LogMessage>(s => s
.Sort(so => so
.Ascending(f => f.Fields["LogEventCategory"].Suffix("keyword"))
)
);
Upvotes: 0
Related Questions
- Elastic search Nest C# - How to Ignore casing in Sorting in a generic way
- SortField in NEST 7.0 alpha 2 with Elasticsearch 7.1
- ElasticSearch NEST - GroupBy then OrderBy
- Elastic Nest Passing Dynamic Values into Sort
- NEST Elasticsearch advanced sorting
- NEST Sort by Text field
- Elasticsearch sort by nested object
- NEST c# Query Creating SORT in Foreach - ElasticSearch
- Sort nested object in Elasticsearch
- Elasticsearch - Order search results ASC