CODEWITHSUNDEEP
solrindexing
doumeasse38
doumeasse38

Reputation: 293

Removing Solr duplicate values into multivalued field

My Solr index contains a multivalued field with duplicate values. How can I remove the duplicates ?

Is it possible to overwrite duplicate values into the multivalued field when indexing ?

Thanks

Upvotes: 16

Views: 5999

Answers (7)

6harat
6harat

Reputation: 622

In latest version of solr you can use add-distinct while doing atomic updates to multivalued fields.

add-distinct: Adds the specified values to a multiValued field, only if not already present. May be specified as a single value, or as a list.

(ref: https://lucene.apache.org/solr/guide/8_8/updating-parts-of-documents.html)

Upvotes: 1

lgonzales
lgonzales

Reputation: 111

I am using solrJ to bind documents, and to avoid duplicated values I defined my multivalued field as a HashSet. 

@Field("description")
public Collection<String> description = new HashSet<>();

Upvotes: 1

Jorge Lazo
Jorge Lazo

Reputation: 388

Really late to the party, but the top answer did not work for me in Solr 6.0 for attempting to add a duplicate entry on a multivalued field. it was missing a processor right before UniqFieldsUpdateProcessorFactory. So adding something like this to my solrconfig.xml worked:

<updateRequestProcessorChain name="uniq-fields">
<processor class="org.apache.solr.update.processor.DistributedUpdateProcessorFactory"/>
<processor class="org.apache.solr.update.processor.UniqFieldsUpdateProcessorFactory">
  <str name="fieldName">YourFieldA</str>
  <str name="fieldName">yourFieldB</str>
</processor>
<processor class="solr.RunUpdateProcessorFactory" />

Where YourFieldA and YourFieldB are defined fields in your schema.xml. Note that you must also add this to the proper requestHandler ie:

  <requestHandler name="/update" class="solr.UpdateRequestHandler" >
<lst name="defaults">
  <str name="update.chain">uniq-fields</str>
</lst>

This will not only prevent duplicates from being added, but also remove all duplicates from your index upon update for the specified fields.

Upvotes: 6

bigGuy
bigGuy

Reputation: 1744

This configuration works for Solr 5.3.1

<updateRequestProcessorChain name="distinct-values" default="true">
    <processor class="solr.DistributedUpdateProcessorFactory"/>
    <processor class="solr.UniqFieldsUpdateProcessorFactory">
        <str name="fieldName">field1</str>
        <str name="fieldName">field2</str>
    </processor>
    <processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>

Upvotes: 0

Persimmonium
Persimmonium

Reputation: 15791

Or you could handle it in Solr, but in an UpdateRequestProcessor so that it happens before indexing and you don't need to learn about analysis chain.

You can use java or a number of scripting languages with the ScriptUpdateProcessor

Upvotes: 0

pranay
pranay

Reputation: 474

I was struggling to accomplish the same. This worked for me. Add the below processor to your solrconfig.xml

<updateRequestProcessorChain name="deduplicateMultiValued" default="true">
        <processor class="org.apache.solr.update.processor.UniqFieldsUpdateProcessorFactory">
            <lst name="fields">
                <str>multivaluedFieldXYZ</str>
            </lst>
        </processor>
        <processor class="solr.RunUpdateProcessorFactory" />
 </updateRequestProcessorChain>

Upvotes: 5

Jayendra
Jayendra

Reputation: 52779

You would need to handle it on the Client side to remove the duplicate values.

You can customize the implementation like RemoveDuplicatesTokenFilterFactory (works for same text at same position) to filter out the tokens. Write an extension basically. OR

Also, If using the multivalued field for just faceting ,the value in faceted field is counted just once. So even if you add multiple same values, that would be reflected as a single value in the facet count entry. Have tested this. you too can confirm.

However, the duplicate values would cause the change in the lengthNorm and hence can have an effect on the scoring.

Upvotes: 1

Related Questions