CODEWITHSUNDEEP
solr
dorgan
dorgan

Reputation: 135

SOLR Indexing min term size

I have my "text" fields defined as follows:

<fieldType name="text" class="solr.TextField" positionIncrementGap="100" omitNorms="false">
<analyzer type="index">
    <tokenizer class="solr.NGramTokenizerFactory" minGramSize="3" maxGramSize="15" />
    <!-- <tokenizer class="solr.WhitespaceTokenizerFactory"/> -->
    <!-- in this example, we will only use synonyms at query time -->
    <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true" />
    <charFilter class="solr.HTMLStripCharFilterFactory" />
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" />
    <filter class="solr.LowerCaseFilterFactory" />
    <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt" />
    <filter class="solr.RemoveDuplicatesTokenFilterFactory" />
</analyzer>
<analyzer type="query">
    <tokenizer class="solr.WhitespaceTokenizerFactory" />
    <charFilter class="solr.HTMLStripCharFilterFactory" />
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" />
    <filter class="solr.LowerCaseFilterFactory" />
    <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt" />
    <filter class="solr.RemoveDuplicatesTokenFilterFactory" />
</analyzer>

When I use the Schema Browsers through the solr admin it seems to be indexing very tiny fragments of this field. I am seeing terms in there such as p s n and so on...

How do i fix this?

Upvotes: 0

Views: 2018

Answers (1)

d whelan
d whelan

Reputation: 804

If I had to guess I would say the small terms are coming from the WordDelimiterFilterFactory

example, terms like "j2ee" would be split up and stored as three terms "j", "2", and "ee"

http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.WordDelimiterFilterFactory

Upvotes: 1

Related Questions