CODEWITHSUNDEEP
lucenefull-text-searchhibernate-search
Tato
Tato

Reputation: 195

Apache Lucene search with stop words

I have search many times but cant fix my issue, here is my code:

   BooleanJunction<BooleanJunction> booleanJunction = queryBuilder.bool();

    String[] fields = new String[]{"field1","field2","field2"};
    MultiFieldQueryParser mfqp = new MultiFieldQueryParser(fields,new StandardAnalyzer(CharArraySet.EMPTY_SET));
    for (String token : tokens) {
        booleanJunction.should(mfqp.parse(token + "*"));
    }

in this case there is not exception, when token is stopword (and, an, etc.), but it does't search exact keywords.

E.g, if there is indexed "andtest", it returns this item, but if there is "and test" it does't search this item.

Could you give me an advice, what's wrong here?

Upvotes: 1

Views: 1548

Answers (1)

yrodiere
yrodiere

Reputation: 9977

When you index "and test", the StandardAnalyzer (which is the default) will tokenize this as two different terms: "and" and "test". Thus searching for "and test*" will not match a document with "and test" in it.

The usual method to match multiple terms in a document would be to use phrase queries, but those don't support wildcards.

If you want to solve your specific problem, and make "and*" match "and test" with a wildcard, a custom analyzer with a shingle filter may be the way to go. I never used it myself, but you can give it a try.

You should probably look into this resource for more information about analysis concepts (here in particular). It's Solr documentation, so some concepts are irrelevant to you (in particular the XML snippets), but most high-level concepts (tokenization, filtering, stemming, ...) are still relevant. And most importantly, this page lists a bunch of analyzers/tokenizers/filters, and those can all be used in Hibernate Search/Lucene.

For everything related to configuring analyzers in Hibernate Search, have a look here

On a side note, if you're adding the wildcard ("*") yourself, it probably means that you should use a custom analyzer with an edgeNGramFilter instead. But it's a bit more complex, since you'll have to use different analyzers when indexing and when querying. So wildcard should be fine for now :)

Upvotes: 1

Related Questions