CODEWITHSUNDEEP
lucenelucene.net
Ted Nyberg
Ted Nyberg

Reputation: 7411

Search for two-letter words in Lucene

I'm trying to find documents containing the acronym "IT".

I've tried searching using the StandardAnalyzer, SimpleAnalyzer and KeywordAnalyzer - same result (no hits whatsoever).

As far as I can see, "it" isn't part of the default stop words?

I can find the documents using a wildcard search, so I know they're in the index.

Any help is greatly appreciated! Cheers!

Upvotes: 0

Views: 1108

Answers (2)

femtoRgon
femtoRgon

Reputation: 33351

The default stopword set does include the word "it". It is defined in StopAnalyzer, and it is:

final List<String> stopWords = Arrays.asList(
   "a", "an", "and", "are", "as", "at", "be", "but", "by",
   "for", "if", "in", "into", "is", "it",
   "no", "not", "of", "on", "or", "such",
   "that", "the", "their", "then", "there", "these",
   "they", "this", "to", "was", "will", "with"
 );

Neither SimpleAnalyzer nor KeywordAnalyzer use stopwords, so these didn't work due to some other issue, possibly a misunderstanding of how they tokenize, or a disagreement between index and query time analyzers.

Upvotes: 3

Ted Nyberg
Ted Nyberg

Reputation: 7411

I tried re-indexing without any stop words...

new IndexWriter(directory,
                new StandardAnalyzer(Version.LUCENE_30, new HashSet<string>()), // No stop words
                true,
                IndexWriter.MaxFieldLength.UNLIMITED);

...and after that I was able to search for "it" as long as I used the same type of analyzer (without any stop words) for searching:

new StandardAnalyzer(Version.LUCENE_30, new HashSet<string>()

Upvotes: 2

Related Questions