Elasticsearch mappings generated differently for different deployments for my spring project

Question

In my spring project, I use annotations to define elasticsearch fields. As far as I understand, elasticsearch mappings are generated on startup of the app based on those annotations. However, mappings generated are different for different deployments.

I am running spring project with Java 8 and ElasticSearch 5.5.0.

Annotations:

@Size(min = MINIMUM_LENGTH_NAME, max = MAXIMUM_LENGTH_NAME)
@NotNull
@Column(name = "name", nullable = false)
@Field(store = Store.YES)
@Field(name = "name_forSort", normalizer = @Normalizer(definition = "lowercase"))
@SortableField(forField = "name_forSort")
private String name;

Expected mappings for name:

{
  "mappings": {
    "properties": {
      "name" : {
        "type" : "text",
        "store" : true
      },
      "name_forSort" : {
        "type" : "keyword",
        "norms" : true,
        "normalizer" : "lowercase"
      }
    }
  }
}

Mappings for the problematic deployment:

{
  "mappings": {
    "properties": {
       "name" : {
         "type" : "text",
         "fields" : {
           "keyword" : {
             "type" : "keyword",
             "ignore_above" : 256
           }
         }
       },
       "name_forSort" : {
         "type" : "text",
         "fields" : {
           "keyword" : {
             "type" : "keyword",
             "ignore_above" : 256
           }
         }
       },
    }
  }
}

Answer 1

The problem was solved by initiating a new ES instance followed by a reindex. It seems the weird mapping was caused by a previously failed reindex with corrupted data.

Answer 2

Elasticsearch 5.5.0 is old, and you did not specify, which Spring Data Elasticsearch (which I will call SDE for brevity) version you rare using. The solution I show in this answer is tested with the current SDE 3.2.0.RC1 which supports Elasticsearch 6.8.2, but as fa as could see in the source code of SDE, this functionality is available in old versions and should work with your version as well.

In your code there are problems:

• You are mixing different annotations that you put on the property, but only the second @Field annotation comes from SDE. @Field(store = Store.YES) is not from SDE, the correct annotation has a store attribute that is of type boolean. Neither @Column nor @SortableField are known to SDE and will be ignored when building the mapping.

• The correct way to use a field multiple times in an index with different settings is by using fields mapping (as you see in the link this was already in 5.5). This is supported by SDE with the @MultiField annotation.

I have run a minimal example with the following files:

normalizer.json

{
    "index": {
        "analysis": {
            "normalizer": {
                "lowercase": {
                    "type": "custom",
                    "char_filter": [],
                    "filter": [ "lowercase" ]
                }
            }
        }
    }
}

Person.java

package com.sothawo.springdataelastictest;

import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldType;
import org.springframework.data.elasticsearch.annotations.InnerField;
import org.springframework.data.elasticsearch.annotations.MultiField;
import org.springframework.data.elasticsearch.annotations.Setting;

@Document(indexName = "person-test")
@Setting(settingPath = "normalizer.json")
public class Person {
    @Id private Long id;

    @MultiField(mainField = @Field(store = true, type = FieldType.Text),
                otherFields = {
                    @InnerField(suffix = "forSort", 
                                normalizer = "lowercase", 
                                type = FieldType.Keyword)
                })
    private String name;

    public Long getId() {
        return id;
    }

    public void setId(Long id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }
}

PersonRepository.java

package com.sothawo.springdataelastictest;

import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;

public interface PersonRepository extends ElasticsearchRepository<Person, Long> {}

Starting up a Spring Boot application with these classes creates the following index:

{
  "person-test": {
    "aliases": {},
    "mappings": {
      "person": {
        "properties": {
          "name": {
            "type": "text",
            "store": true,
            "fields": {
              "forSort": {
                "type": "keyword",
                "normalizer": "lowercase"
              }
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "number_of_shards": "5",
        "provided_name": "person-test",
        "creation_date": "1563903601344",
        "analysis": {
          "normalizer": {
            "lowercase": {
              "filter": [
                "lowercase"
              ],
              "type": "custom",
              "char_filter": []
            }
          }
        },
        "number_of_replicas": "1",
        "uuid": "WhSFtV-xQ8acXxX6R0gLyg",
        "version": {
          "created": "6070299"
        }
      }
    }
  }
}