Hibernate Search 6: Trying to sort dynamic fields with field templates leads to error 'unknown field'

Question

I'm trying to switch from Hibernate 5 to Hibernate 6 with Elasticsearch backend. In our project there is an entity with dynamic number of fields. As I understand it the best way to handle this would be to use dynamic fields with templates as described here https://docs.jboss.org/hibernate/search/6.2/reference/en-US/html_single/#bridge-index-field-dsl-dynamic

I created a property binder to add the dynamic fields to the index. This works well, the fields are added to the index and can be searched. The problem is now how can I sort this fields? I'm getting the error: org.hibernate.search.util.common.SearchException: HSEARCH000610: Unknown field 'customFields.town_TextForSort.keyword'.

The property binder contains the following code:

public class AddressElementBinder implements PropertyBinder {

    @Override
    public void bind(PropertyBindingContext context) {
        context
            .dependencies()
            .useRootOnly();

        IndexSchemaElement schemaElement = context.indexSchemaElement();
        
        IndexSchemaObjectField customFields = schemaElement
                .objectField( "customFields", ObjectStructure.FLATTENED)
                .multiValued();

        customFields.fieldTemplate( 
                "fieldValueTemplate_TextDefault", 
                f -> f.asString()
                .analyzer("customAnalyzer")
                .searchable( Searchable.YES )
                .sortable(Sortable.NO)
        )
        .matchingPathGlob( "*_TextDefault" );

        customFields.fieldTemplate( 
                "fieldValueTemplate_TextForSort",
                f -> f.asString()
                .normalizer("customNormalizer")
                .sortable( Sortable.YES )
                .searchable( Searchable.NO )
        )
        .matchingPathGlob( "*_TextForSort" );

        context.bridge( Set.class, new AddressElementBridge(customFields.toReference()) ); 
    }
}

The property bridge:

public class AddressElementBridge implements PropertyBridge<Set> {

    private IndexObjectFieldReference customFieldFieldReference = null;
    
    public AddressElementBridge(IndexObjectFieldReference customFieldFieldReference) {
        this.customFieldFieldReference = customFieldFieldReference;
    }
    
    @Override
    public void write(DocumentElement target, Set bridgedElement, PropertyBridgeWriteContext context) {
        if(bridgedElement == null || bridgedElement.isEmpty() || this.customFieldFieldReference == null) {
            return;
        }
                                                
        DocumentElement customFieldElement = target.addObject( this.customFieldFieldReference ); 
                
        AbstractPostalAddress postalAddress = ContactAddressHelper.getMainPostalAddress(bridgedElement);        
        List<CustomField> fields = postalAddress.getFields(context);
        if(fields == null ||  fields.isEmpty()) {
            return;
        }
        
        for (FieldInfoFullTextIndex customField : fields) {
            
            String fieldName = customField.getFieldName();
            Object fieldValue = customField.getFieldValue();
            
            if(fieldValue != null) {
                                
                if(fieldValue instanceof String) {          
                    
                    fieldName = fieldName + "_TextDefault";
                    customFieldElement.addValue( fieldName, fieldValue );

                    fieldName = fieldName + "_TextForSort";
                    customFieldElement.addValue( fieldName, fieldValue );
                    
                }
                
            }
        }
        
    }

The new fields are added to the index like this:

{
    "mappings": {
        "_doc": {
            "_source": {
                "enabled": false
            },
            "properties": {
                "_entity_type": {
                    "type": "keyword",
                    "index": false
                },
                "customFields": {
                    "dynamic": "true",
                    "properties": {
                        "town_TextDefault": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "town_TextForSort": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

Analyzer and normalizer definition:

public class GlobalElasticsearchAnalysisConfigurer implements ElasticsearchAnalysisConfigurer {

    @Override
    public void configure(ElasticsearchAnalysisConfigurationContext context) {
    
        context.analyzer( "customAnalyzer" )
            .custom()
            .tokenizer( "standard" ) 
            .charFilters( "html_strip" ) 
            .tokenFilters( "lowercase", "asciifolding" );

        context.normalizer( "customNormalizer" )
            .custom() 
            .tokenFilters( "lowercase", "asciifolding" );

    }
}

Using path "customFields.town_TextForSort" for sorting is not a good idea because it is of type "text" and not "keyword". Using path "customFields.town_TextForSort.keyword" for sorting does not work because of the error: HSEARCH000610: Unknown field 'customFields.town_TextForSort.keyword'.

if a run a search using the Elasticsearch interface directly with path "customFields.town_TextForSort.keyword", then it works. Elasticsearch returns a sorted result.

Is this is a bug in Hibernate Search 6? Because the field validation fails (Hibernate Search does not know about the sub field "keyword". Or is my setup wrong? I assumed that the fields would be all of type keyword and not text if ObjectStructure.FLATTENED is used. Or is Elasticsearch creating a keyword sub field? How can I force the fields to be of type keyword instead of text?

Update 2023-03-22: I updated the code blocks above.

Answer 1

First, Hibernate Search does not support multi-fields yet (see HSEARCH-3465), so this .keyword sub-field is not something it can work with.

Second, even if it did, that .keyword subfield you see in your Elasticsearch schema is never mentioned in the field definition on the Hibernate Search side: it's added automatically by Elasticsearch because of some default template on the server side. So Hibernate Search wouldn't know about this subfield.

The solution would be for you to add a sibling field instead of relying on multi-fields. And that's what you apparently tried to do... but somehow the Elasticsearch schema doesn't match what you configured on the Hibernate Search side:

1. I don't see "customFields" mentioned anywhere in your Elasticsearch schema.
2. I don't see your custom analyzer and normalizer mentioned anywhere in your Elasticsearch schema.

So there's something wrong.

I think you either:

1. Forgot to have Hibernate Search create the Elasticsearch schema before performing your tests.
2. Forgot to create the customFields object field in your bridge (e.g. DocumentElement customFields = target.addObject("customFields")
3. Mistakenly added the value of fieldValueTemplate_TextDefault/fieldValueTemplate_TextForSort to the root (the target parameter passed to your field) instead of adding them to your customFields object field.

I'd need the full code of your binder/bridge and your full Elasticsearch schema to say for sure.