CODEWITHSUNDEEP
javalucenehibernate-search
Simmer
Simmer

Reputation: 163

Why is Lucene sometimes not matching InChIKeys?

I have indexed my database using Hibernate Search. I use a custom analyzer, both for indexing and for querying. I have a field called inchikey that should not get tokenized. Example values are:

When I look into my index with Luke I can confirm that they are not tokenized, as required.

However, when I try to search them using the web app, some inchikeys are found and others are not. Curiously, for these inchikeys the search DOES work when I search without the last hyphen, as so: BBBAWACESCACAP-UHFFFAOYSA N

I have not been able to find a common element in the inchikeys that are not found.

Any idea what is going on here?

I use a MultiFieldQueryParser to search over the different fields in the database:

    String[] searchfields = Compound.getSearchfields();
    MultiFieldQueryParser parser = new MultiFieldQueryParser(Version.LUCENE_29, Compound.getSearchfields(), new ChemicalNameAnalyzer());
    //Disable the following if search performance is too slow
    parser.setAllowLeadingWildcard(true);
    FullTextQuery fullTextQuery = fullTextSession.createFullTextQuery(parser.parse("searchterms"), Compound.class);
    List<Compound> hits = fullTextQuery.list();

More details about our setup have been posted here by Tim and I.

Upvotes: 2

Views: 211

Answers (1)

Simmer
Simmer

Reputation: 163

It turns out the last entries in the input file are not being indexed correctly. These ARE being tokenized. In fact, it seems they are indexed twice: once without being tokenized and once with. When I search I cannot find the un-tokenized.

I have not yet found the reason, but I think it perhaps has to do with our parser ending while Lucene is still indexing the last entries, and as a result Lucene reverting to the default analyzer (StandardAnalyzer). When I find the culprit I will report back here.

Adding @Analyzer(impl = ChemicalNameAnalyzer.class) to the fields solves the problem, but what I want is my original setup, with the default analyzer defined once, in config, like so:

<property name="hibernate.search.analyzer">path.to.ChemicalNameAnalyzer</property>

Upvotes: 2

Related Questions