CODEWITHSUNDEEP
solrsolr7
blankCoder
blankCoder

Reputation: 172

Solr 7 Upgrade - /update requests going to only 1 node out of 3

We recently upgraded from Solr 6.6 to Solr 7.7 (cluster with 3 nodes) and are facing a peculiar issue. We index around 9-10 million docs every monday to Solr as part of a batch process and this process used to take around 50-55 mins to complete with Solr6.6. After migration to Solr 7.7 with all configuration (solrconfig.xml and schema) exactly same as existing collection on Solr 6.6 we observed the same process is taking 5-6 hrs to index the similar amount of load. On checking solr logs we found that most of the update requests (>90%) went to 1 of the nodes out of 3. The solr cluster is fronted by a GCP load balancer which is configured to have a round robin policy.

We are using compositeId as router with no prefix based routing. Exampe doc: {id:AB_111_1_1, qty:10.0, name:111, part:1, date:2020-06-29T00:00:00Z }

Solr7 cluster is enabled with auth plugin which was not enabled on Solr 6 cluster. There was no other change from application/process side for this behaviour. We are not using any solr client (solrs or solrj) to send the update/query requests to solr. It is a normal Http Connection to load balancer with auth enabled.

This behavior degrades with time, and indexing becomes much more slower. We observed similar behaviour when we used solr data import handler to copy the documents from solr 6.6 to solr 7.7. Initially the index speed was around 2K docs per sec but in 10-15 mins this used to start to get impacted and speed reduced.

Upvotes: 0

Views: 120

Answers (2)

blankCoder
blankCoder

Reputation: 172

The solution to the above problem was upgrading the infrastructure on which solr was hosted. After upgrading CPU + Memory, the performance was better than Solr 6. This was found out by trial method with different configurations.

Upvotes: 0

petercontrains
petercontrains

Reputation: 1

Verify that apache Tika is the correct version and that the Java memory is allocated correctly. Monitor your operation with htop

Upvotes: 0

Related Questions