CODEWITHSUNDEEP
hibernate-search
Rokko_11
Rokko_11

Reputation: 937

Hibernate Search: Indexing only a few entites of a huge table without loading all entries

I have a huge table which I write into a Lucene Index using Hibernate Search Version 5.11.5. After a (existing) raw SQL import into this table I have to manually update the search index. This SQL import runs multiple times a day and reindexing should not block incoming search requests for more than a few seconds.

On each entity there exists a date field "modified" so I already annotated this @Indexed Entity with an EntityIndexingInterceptor like following:

public class CustomEntityIndexingInterceptor implements EntityIndexingInterceptor<HugeTableEntity> {
   public static Date lastModified = //some logic;

   @Override
   public IndexingOverride onAdd(HugeTableEntity entity) {
      return IndexingOverride.APPLY_DEFAULT;
   }

   @Override
   public IndexingOverride onUpdate(HugeTableEntity entity) {
      if (entity.getModified().after(lastModified)) {
         return IndexingOverride.APPLY_DEFAULT;
      }
      return IndexingOverride.SKIP;
   }
}

This code works, but reindexing takes a lot of time because all entites get loaded. Only < 0.001% match the condition of the modified date.

I saw that deep inside of Hibernate Search there exists a class IdentifierProducer, which loads all IDs in the method loadAllIdentifiers. I would love to add a SQL filter to the inner criteria - something like "where modified > :given_date".

Do you know if I can customize the IdentifierProducer without copying all the code? Do you know another smart way to solve my problem?

Regards

Upvotes: 1

Views: 702

Answers (1)

yrodiere
yrodiere

Reputation: 9977

The MassIndexer cannot be used to index just part of your data... yet. IdentifierProducer is an internal class an you should not try to alter it.

What you can do is run a query to list the entities that were affected by your import, and ask Hibernate Search to reindex them, for example by batches of 20 elements. You can take some inspiration from this example from the documentation. In your case, of course, you will add filters to the Hibernate ORM query, to only go through entities that actually changed because of your import.

Don't forget to remove your indexing interceptor, which should no longer be useful.

Upvotes: 1

Related Questions