CODEWITHSUNDEEP
solrsolrnetsolr4
user2711965
user2711965

Reputation: 1825

Solr to Tokenize on white space, comma and period

I am trying to force Solr to tokenize document on white-space, comma, : and ;. Something similar to what SQL Server Full Text search does. If I use text_general field then it tokenizes on other characters as well like ('/','\','-') , I tried using 

<tokenizer class="solr.PatternTokenizerFactory" pattern="\s*,:;\s*"/>

But it doesn't tokenize it. Here is how my FieldType looks like:

<fieldType name="text_sqlserver" class="solr.TextField" positionIncrementGap="100">
  <analyzer type="index">
    <tokenizer class="solr.PatternTokenizerFactory" pattern="\s*,:;\s*"/>
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.PatternTokenizerFactory" pattern="\s*,:;\s*"/>
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
    <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>

Is there anything that I am missing ? I have to search for case insensitive comparison as well.

Upvotes: 2

Views: 3631

Answers (1)

EricLavault
EricLavault

Reputation: 16035

Your pattern is actually wrong, try something like this instead : pattern="[\s,;:]"

An alternative you might want to try :

Upvotes: 6

Related Questions