CODEWITHSUNDEEP
solrsolr4
Avner Levy
Avner Levy

Reputation: 6751

How to update Solr documents on the Solr server side with custom handler / plugin

I have a core with millions of records.
I want to add a custom handler which scan the existing documents and update one of the field based on a condition (age>12 for example).
I prefer doing it on the Solr server side for avoiding sending millions of documents to the client and back.
I was thinking of writing a solr plugin which will receive a query and update some fields on the query documents (like the delete by query handler).
I was wondering whether there are existing solutions or better alternatives.
I was searching the web for a while and couldn't find examples of Solr plugins which update documents (I don't need to extend the update handler).
I've written a plug-in which use the following code which works fine but isn't as fast as I need.
Currently I do: 

AddUpdateCommand addUpdateCommand = new AddUpdateCommand(solrQueryRequest); 
DocIterator iterator = docList.iterator(); 
SolrIndexSearcher indexReader = solrQueryRequest.getSearcher(); 
while (iterator.hasNext()) { 
   Document document = indexReader.doc(iterator.nextDoc()); 
   SolrInputDocument solrInputDocument = new SolrInputDocument(); 
   addUpdateCommand.clear(); 
   addUpdateCommand.solrDoc = solrInputDocument; 
   addUpdateCommand.solrDoc.setField("id", document.get("id")); 
   addUpdateCommand.solrDoc.setField("my_updated_field", new_value); 
   updateRequestProcessor.processAdd(addUpdateCommand); 
}

But this is very expensive since the update handler will fetch again the document which I already hold at hand.
Is there a safe way to update the lucene document and write it back while taking into account all the Solr related code such as caches, extra solr logic, etc?
I was thinking of converting it to a SolrInputDocument and then just add the document through Solr but I need first to convert all fields.
Thanks in advance, Avner

Upvotes: 5

Views: 1680

Answers (2)

cheffe
cheffe

Reputation: 9500

To point out where to put custom logic, I would suggest to have a look at the SolrEntityProcessor in conjunction with Solr's ScriptTransformer.

The ScriptTransformer allows to compute each entity after it is extracted from the source of a dataimport, manipulate it and add custom field values before the new entity is written to solr.

A sample data-config.xml could look like this

<?xml version="1.0" encoding="UTF-8" ?>
<dataConfig>

    <script>
    <![CDATA[
        function calculateValue(row)        {
            row.put("CALCULATED_FIELD", "The age is: " + row.get("age"));
            return row;
        }
    ]]>
    </script>

  <document>
    <entity name="sep" processor="SolrEntityProcessor" 
        url="http://localhost:8080/solr/your-core-name" 
        query="*:*"
        wt="javabin"
        transformer="script:calculateValue">
            <field column="ID" name="id" />
            <field column="AGE" name="age" />
            <field column="CALCULATED_FIELD" name="update_field" />
    </entity>
  </document>
</dataConfig>

As you can see, you may perform any data transformation you like and is expressible in javascript. So this would be a good point to express your logic and transformations.

You say one constraint maybe age > 12. I would handle this via the query attribute of the SolrEntityProcessor. You could write query=age:[* TO 12] so that only records with an age up to 12 would be read for the update.

Upvotes: 0

phanin
phanin

Reputation: 5487

I'm not sure whether the following is going to improve the performance, but thought it might help you.

Look at SolrEntityProcessor

Its description sounds very relevant to what you are searching for.

This EntityProcessor imports data from different Solr instances and cores. 
The data is retrieved based on a specified (filter) query. 
This EntityProcessor is useful in cases you want to copy your Solr index 
and slightly want to modify the data in the target index. 
In some cases Solr might be the only place were all data is available.

However, I couldn't find an out-of-the-box feature to embed your logic. So, you may have to extend the following class.

SolrEntityProcessor and the link to sourcecode

You may probably know, but a couple of other points.

1) Make the entire process exploit all the cpu cores available. Make it multi-threaded.

2) Use the latest version of Solr.

3) Experiment with two Solr apps on different machines with minimal network delay. This would be a tough call : 

same machine, two processes VS two machines, more cores, but network overhead.

4) Tweak Solr cache in a way that applies to your use-case and particular implementation.

5) A couple of more resources: Solr Performance Problems and SolrPerformanceFactors

Hope it helps. Let me know the stats despite this answer. I'm curious and your info might help somebody later.

Upvotes: 0

Related Questions