How to sort 'Hibernate with Lucene' search results based on 'whole words' rather than contains

Question

I'm using Hibernate Search to provide full text search of products/items in our store application. Following is how my Item class is :

@Entity
@Table(name = "items", indexes = {
    @Index(name = "idx_item_uuid", columnList = "uuid", unique = true),
    @Index(name = "idx_item_gtin", columnList = "gtin", unique = true),
})
@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
@EqualsAndHashCode(onlyExplicitlyIncluded = true, callSuper = true)
@ToString(exclude = {"storeItems"})
@Indexed
@AnalyzerDef(name = "ngram",
    tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class),
    filters = {
        @TokenFilterDef(factory = StandardFilterFactory.class),
        @TokenFilterDef(factory = LowerCaseFilterFactory.class),
        @TokenFilterDef(factory = StopFilterFactory.class),
        @TokenFilterDef(factory = NGramFilterFactory.class,
            params = {
                @Parameter(name = "minGramSize", value = "1"),
                @Parameter(name = "maxGramSize", value = "3")})
    }
)
public class Item extends BaseModel {

  @Column(nullable = false)
  @Field(analyzer = @Analyzer(definition = "ngram"))
  private String name;

  @OneToMany(orphanRemoval = true, cascade = CascadeType.ALL, mappedBy = "item", fetch = FetchType.EAGER)
  @Fetch(FetchMode.SELECT)
  private List<Image> images;

  @OneToMany(mappedBy = "item", cascade = CascadeType.REFRESH)
  @Fetch(FetchMode.SELECT)
  @JsonIgnore
  @IndexedEmbedded(includePaths = {"store.uuid"})
  private Set<StoreItem> storeItems;

  @Enumerated(EnumType.STRING)
  private QuantityType quantityType;

  @Column(nullable = false, length = 14)
  private String gtin;

  private String articleSize;

  @ManyToOne(fetch = FetchType.EAGER)
  @JoinColumn(name = "brand_id", foreignKey = @ForeignKey(name = "fk_brands_items"))
  private Brand brand;

  private String supplierName;

  @ManyToOne(fetch = FetchType.EAGER)
  @JoinColumn(name = "category_id", foreignKey = @ForeignKey(name = "fk_categories_items"))
  @IndexedEmbedded(includePaths = {"uuid"})
  private Category category;

  private String taxType;

  private Double taxRate;

  @Lob
  private String marketingMessage;

  private boolean seasonal;

  private String seasonCode;

  @Lob
  private String nutritionalInformation;

  @Lob
  private String ingredients;

  private Double depth;

  private String depthUnit;

  private Double height;

  private String heightUnit;

  private Double width;

  private String widthUnit;

  private Double netContent;

  private String netContentUnit;

  private Double grossWeight;

  private String grossWeightUnit;

  private Double maxStorageTemp;

  private Double minStorageTemp;

  private Double maxTransportTemp;

  private Double minTransportTemp;

  private boolean organic;

  private String origin;

}

And following is how my custom repository is searching for the items in a specific store :

  @Override
  public List<Item> findItemBySearchStrAndStoreUuid(final String searchStr, final String storeUuid) {
    final EntityManager entityManager = entityManagerFactory.createEntityManager();

    final FullTextEntityManager manager = Search.getFullTextEntityManager(entityManager);
    entityManager.getTransaction().begin();

    final QueryBuilder qb = manager.getSearchFactory()
        .buildQueryBuilder().forEntity(Item.class).get();

    final Query query = qb.bool()
        .must(qb.keyword().onField("name").matching(searchStr).createQuery())
        .must(qb.keyword().onField("storeItems.store.uuid").matching(storeUuid).createQuery())
        .createQuery();

    return executeQuery(entityManager, manager, query);
  }

We have about 13k items in database and mostly have Swedish names, so when a customer searches for milk in Swedish "mjölk", Milk related items should pop up, they do but sorting is not how we want, for example.

Expected Results :

1. mjölk
2. mjölk chocolate
3. Kokosmjölk

Actual Results :

1. Kokosmjölk
2. mjölk chocolate
3. mjölk

Example might make it seems like I just need to reverse the sorting, but the problem is it's not how actual results really are, they are more random, but the issue is I need Milk come first, then items that has 'Milk' as a whole word, then all those items that has it as a substring'.

So please guide me of how I should enhance my analyzer/query to achieve such sorting, I need to give results even with a single character, search should also handle some typos, Hence I used Ngram filter with above settings.

Also, I did try using SwedishLightStemFilterFactory, that did help a bit, but then items stopped showing up unless someone typed 'mjölk' completely and correctly.

Thanks in advance.

Answer 1

I would consider 2 things:

• ASCIIFoldingFilterFactory: replacing accented characters with the plain one
• Separate analyser for sorting where the value is not tokenized and only lowercased

Sorting in Hibernate typically involves a different strategy.

Answer 2

You need to declare a separate field on the same property, used for sorting exclusively, and assign it a normalizer instead of an analyzer.

See https://docs.jboss.org/hibernate/search/5.11/reference/en-US/html_single/#section-normalizers