Hibernate Search drops and re-creates index for every new data indexed

Question

So I've built an Spring Application that has a Hibernate Search (Version 6.2.2.Final) Module. Through Docker I started an Elasticsearch (Version 8.9.1) Container alongside Enterprise Search and Kibana for management.

I am trying to index this Entity:

@Entity
@Indexed(index = "search-fileentity-new")
public class FileEntityHS {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private long id;
    @KeywordField
    private String fileIdAsString;
    @Transient
    private transient long fileId;
    @KeywordField
    private String userIdAsString;
    @Transient
    private transient long userId;
    @FullTextField
    private String customName;
    @ElementCollection
    private List<String> tags;
    @FullTextField()
    private String content;
    @KeywordField
    private String fileGuid;

    public FileEntityHS() {
        this.fileId = 0L; // oder andere Default-Werte
        this.userId = 0L;
        this.fileIdAsString = "0";
        this.userIdAsString = "0";
    }

    public FileEntityHS(long fileId, long userId, String customName, List<String> tags, String content, String fileGuid) {
        this.fileId = fileId;
        this.userId = userId;
        this.customName = customName;
        this.tags = tags;
        this.content = content;
        this.fileGuid = fileGuid;

        this.fileIdAsString = Long.toString(this.fileId);
        this.userIdAsString = Long.toString(this.userId);
    }
}

And this is the Code I am trying to index with:

@Service
public class SearchServiceImpl implements SearchService {

    private final EntityManager entityManager;
    private final SearchSession searchSession;

    public SearchServiceImpl(EntityManager entityManager) {
        this.entityManager = entityManager;
        this.searchSession = Search.session(entityManager);
    }

    @Override
    @Transactional
    public short indexFile(FileEntityHS file) {
        try {
            searchSession.indexingPlan().addOrUpdate(file);
        } catch(Exception e) {
            e.printStackTrace();
        }
        return 0;
    }
}

In my Elasticsearch instance I have created an Index with the name thats specified in my Entity search-fileentity-new. For some reason Hibernate Search always re-creates an Index with -000001 at the end. For example search-fileentity-new-000001.

So the index with the last indexed document is deleted and a new index with the same name and with the indexed File is created. This is not expacted behaviour for me as documentation does not mention anything about this.

This is part of my pom.xml:

   <properties>
        <hibernate.version>6.2.2.Final</hibernate.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
        </dependency>
        <dependency>
            <groupId>org.hibernate.orm</groupId>
            <artifactId>hibernate-core</artifactId>
            <version>${hibernate.version}</version>
        </dependency>
        <dependency>
            <groupId>org.hibernate.search</groupId>
            <artifactId>hibernate-search-mapper-orm-orm6</artifactId>
            <version>${hibernate.version}</version>
        </dependency>
        <dependency>
            <groupId>org.hibernate.search</groupId>
            <artifactId>hibernate-search-backend-elasticsearch</artifactId>
            <version>${hibernate.version}</version>
        </dependency>
    </dependencies>

I looked a bit around and found some things I've tried. In my .properties I have this:

# Hibernate Search
spring.jpa.properties.hibernate.search.backend.type=elasticsearch
hibernate.search.backend.type=elasticsearch

spring.jpa.properties.hibernate.search.backend.index_defaults.lifecycle.strategy=none
hibernate.search.backend.index_defaults.lifecycle.strategy=none
spring.jpa.properties.hibernate.search.index_schema_management.strategy=none
hibernate.search.index_schema_management.strategy=none
spring.jpa.properties.hibernate.search.schema_management.strategy=none
hibernate.search.schema_management.strategy=none

I thought maybe it's some kind lifecycle setting so I tried to disable everything in the .properties but it doesn't do anything.

Next I looked in Elasticsearch and there weren't any settings that suggested a lifecycle control in Elasticsearch for that index. But maybe I looked at the wrong place.

I've also stumbled upon another way to index documents with a MassIndexer:

SearchSession searchSession = Search.session(entityManager);
MassIndexer massIndexer = searchSession.massIndexer(FileEntityHS.class);
massIndexer.startAndWait();

But when I tried that Hibernate Search tried to persist the document in a Mysql Database. So that didn't work either.

Update 1
Okay, so one of the problems I mentioned was that the Index always gets recreated because it looked like it to me. But I noticed that this is not true in Fact the data in the Index gets overriten. As for indexnaming difference the property hibernate.search.backend.layout.strategy=simple (default) causes this behavior. I changed it to no-alias and now the specified Index in the entity is getting used.

But the problem that with every new

SearchSession.indexingPlan().addOrUpdate(Entity);

my data getting lost was still present.

Update 2
I was able to fix the overriding of the data. First I noticed that the id of the documents indexed is always 0. So I was lookig at how to define the id in the Entity. I found that you would not use Long as type instead you would use String.

That got me one step further. Now I got an error that stated that the length of the id was null. For that to get fixed I had to manuelly set the id for my documents.

...
import java.util.UUID;  
  
@Entity  
@Indexed(index = "search-fileentity-new")  
public class FileEntityHS {  
  
    @Id  
    private String id;  
    ...
  
    public FileEntityHS() {  
        this.id = UUID.randomUUID().toString();  
        ...
    }  
  
    public FileEntityHS( ..., String fileGuid) {
        this.id = fileGuid;  
        ...
    }  

    // getter and setter

So now data is getting indexed seperatly with there own DocumentId.

Note: Thanks to all People that shared their knowledge!

Answer 1

In your case, I would recommend to go step by step and find the suitable solution, because if you try to change all at once, you can't find the exact solution. Also try the following solutions as well:

• You've set the index_schema_management_strategy: none try to change it to index_schema_management_strategy: drop-and-create. It is the property related to Hibernate Search configuration when using Elasticsearch as the backend. After change, Hibernate Search will drop the existing ES index and creates new one when the applications starts. As the reference you can have a look at the following: Link

• Also try to override the Index name:

  @Entity
  @Indexed(index="search-fileentity-new")
  @IndexingOverride(name="YOUR_INDEX_NAME")
  public class FileEntityHS{
  
         // rest code       
    }
  

• As the additional source you can look following: How to configure index names for multiple Environments? - which also discusses similar issue.

• Following is also useful link which shows the example of using Elasticsearch native type.

Answer 2

First, I'd recommend you start simple. You don't need to create the Elasticsearch index yourself, Hibernate Search can do it. Just remove all configuration and follow the getting started guide. Once that works, try something more complex.

Second, if you do want to create the index manually:

1. Use the schema management strategy validate, not none. This will tell you exactly what is wrong on startup, instead of leaving you in the dark like this. See https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#mapper-orm-schema-management-strategy
2. You must make sure your index's name and aliases match what Hibernate Search expects. In your case, your index should have aliases search-fileentity-new-write and search-fileentity-new-read. See https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#backend-elasticsearch-indexlayout

But when I tried that Hibernate Search tried to persist the document in a Mysql Database. So that didn't work either.

I'm sorry but this doesn't make sense. Hibernate Search never persists documents in a database, the code to do that simply isn't there.

Judging from the strange behavior you're experiencing, if I were you I'd check that you're not starting an EntityManagerFactory manually somewhere. You're not supposed to do that, Spring should do that.