uniqueKey generation does not work in SolrCloud (but works if standalone)

Question

I have faced a weird issue when the same application code fails indexing a document without a unique key (should be auto-generated by SOLR) in SolrCloud and succeeds indexing it in standalone SOLR instance (or even in cloud mode, but from web interface of one of the replicas). Difference is obviously only between clients (CloudSolrClient vs HttpSolrClient) and SOLR URLs (Zokeeper hostname+port vs standalone SOLR instance hostname and port).

I am using SOLR 5.1. In cloud mode I have 1 shard and 3 replicas. Documentation states:

Schema defaults and copyFields cannot be used to populate the uniqueKey field. You can use UUIDUpdateProcessorFactory to have uniqueKey values generated automatically.

Therefore I have added my uniqueKey field to the schema:

<fieldType name="uuid" class="solr.UUIDField" indexed="true" />
...
<field name="id" type="uuid" indexed="true" stored="true" required="true" />
...
<uniqueKey>id</uniqueKey>

Then I have added updateRequestProcessorChain to my solrconfig:

<updateRequestProcessorChain name="uuid">
    <processor class="solr.UUIDUpdateProcessorFactory">
        <str name="fieldName">id</str>
    </processor>
    <processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>

And made it the default for the UpdateRequestHandler:

<initParams path="/update/**">
 <lst name="defaults">
  <str name="update.chain">uuid</str>
 </lst>
</initParams>

Adding new documents with null/absent id works fine as from web-interface of one of the replicas, as when using SOLR in standalone mode (non-cloud) from my application. Although when only I'm using SolrCloud and add document from my application (using CloudSolrClient from SolrJ) it fails with "org.apache.solr.client.solrj.SolrServerException: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Document is missing mandatory uniqueKey field: id"

All other operations like ping or search for documents work fine in either mode (standalone or cloud).

Did anyone experience the same behavior? What may be a solution here?

INVESTIGATION (i.e. more details):

1. In standalone mode obviously update request is:

   POST standalone_host:port/solr/collection_name/update?wt=json
   

2. In SOLR cloud mode, when adding document from one replica's web interface, update request is (found through inspecting the call made by web interface):

   POST replica_host:port/solr/collection_name_shard1_replica_1/update?wt=json
   

In both these cases payload is something like:

{
    "add": {
        "doc": {
                 .....
        },
        "boost": 1.0,
        "overwrite": true,
        "commitWithin": 1000
    }
}

In case when CloudSolrClient is used, the following happens (found through debugging):

1. Using ZK and some logic, URL list of replicas is constructed that looks like this:

   [http://replica_1_host:port/solr/collection_name/,
    http://replica_2_host:port/solr/collection_name/,
    http://replica_3_host:port/solr/collection_name/]
   

2. This code is called:

   LBHttpSolrClient.Req req = new LBHttpSolrClient.Req(request, theUrlList);
   LBHttpSolrClient.Rsp rsp = lbClient.request(req);
   return rsp.getResponse();
   

   Where the second line fails with the exception.

If to debug the second line further, it ends up calling HttpClient.execute (from HttpSolrClient.executeMethod) for:

POST http://replica_1_host:port/solr/collection_name/update?wt=javabin&version=2 HTTP/1.1
POST http://replica_2_host:port/solr/collection_name/update?wt=javabin&version=2 HTTP/1.1
POST http://replica_3_host:port/solr/collection_name/update?wt=javabin&version=2 HTTP/1.1

And the very first request returns 400 Bad Request with replica 1 logging "Document is missing mandatory uniqueKey field: id" in the logs.

The funny thing is that when I execute the same request using POSTMAN (but with JSON instead of binary payload), it works! Am I doing something wrong here? I assume it's definitely something in the way of how the request is made...

UPDATE:

I have used local proxy in order to see the difference in these 2 requests sent by my application in order to understand what is different there. Looks like the only difference is content type. In case of cloud mode the payload for POSTing document is sent as "application/javabin" while in standalone mode it's sent as "application/xml; charset=UTF-8". Everything else is the same. First request results in 400 while second is 200. I think this may be a SolrJ/SOLR bug, therefore filed a ticket for that. Will keep this thread updated.

Answer 1

First of all, it worked in Standalone mode since HttpSolrClient in version 5.1 sends payload as XML (excluding null-values - this part is important) while CloudSolrClient sends it as "application/javabin" in binary serialized format (with nulls included). In version 6.2 both of these send payload as "application/javabin" in binary serialized format, hence this issue appears for both Cloud and Standalone mode in version 6.2.

The real cause of the issue is that in order for UUID to be generated, the field must be absent at all. If it's present and it's null, UUID generation is skipped. That's the reason of the error. So, if we're using SolrInputDocument to index our documents, then it's easy - we simply should not add value for "id" field. But what if we're using POJOs with "org.apache.solr.client.solrj.beans.Field" annotations? We can't exclude a field from there. That's where IgnoreFieldUpdateProcessorFactory comes into play:

<updateRequestProcessorChain name="uuid">
    <!-- Using IgnoreFieldUpdateProcessorFactory because of https://issues.apache.org/jira/browse/SOLR-9493:
         can't generate UUID for a field coming as NULL, field must be absent. -->
    <processor class="solr.IgnoreFieldUpdateProcessorFactory">
        <str name="fieldName">id</str>
    </processor>
    <processor class="solr.UUIDUpdateProcessorFactory">
        <str name="fieldName">id</str>
    </processor>
    <processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>

Having this in solrconfig.xml and using "uuid" updateRequestProcessorChain as shown in the question above, everything works like a charm ("id" field is removed from the document before UUID generation). Although in this case it's impossible to add documents with custom IDs, but that's something that remains as another question for another thread.