CODEWITHSUNDEEP
javasolr
Mantu Singh
Mantu Singh

Reputation: 252

How to get no of documents before commit of full-import in solr?

In my solr data full-import, database sometime returns 0 records due to some reason. I want to put a check if database return 0 record rollback the index. I don't want to commit it. Any help will be appreciated.

Upvotes: 2

Views: 596

Answers (2)

Shishir Kumar
Shishir Kumar

Reputation: 8201

SIMPLE ANSWER: If there is actually a need to have a check before starting the data import handler, then there should be a custom implementations of DataImportHandler & DataImporter configured in solrconfig.xml

EXPLAINATION:

Custom MyDataImportHandler class should look something like below:

MyDataImportHandler.class

class MyDataImportHandler extends DataImportHandler {
    :
    private MyDataImporter myImporter;
    :
    :

    @Override
    public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp) throws Exception {
        :
        :
        if(command != null && DataImporter.FULL_IMPORT_CMD.equals(command)) {
            :
            :
            // Check out DataImportHandler.handleRequestBody() for more details 
            myImporter.runCmd(requestParams, sw);
            :
        } else {
            super.handleRequestBody(req, rsp);
        }
        :
        :
    }

    :
    :
    :
}

MyDataImporter.class

class MyDataImporter extends DataImporter {
    :
    :
    // Check DataImporter.doFullImport() for more details
    @Override
    public void doFullImport(SolrWriter writer, RequestInfo requestParams) {
        :
        :
        // Check if count of documents in equal to 0, then throw exception
        if(docBuilder.importStatistics.docCount==0) {
            docBuilder.rollback();
            throw new Exception ("Count of documents is 0");
        }
        :
        :
    }
    :
}

solrconfig.xml

<requestHandler name="/dataimport"
        class="foo.bar.MyDataImportHandler">
        <lst name="defaults">
            <str name="config">data-config.xml</str>
        </lst>
    </requestHandler>

On a side note, before implementing this solution, examine the need of it. I strongly believe that you may not be required to do all this. If there is any case of connection break/reset in database, then the Solr's default DIH will throw an Exception and subsequently rollback the index to the last successfully indexed documents.

Hope this helps you.

References:

  1. org.apache.solr.handler.dataimport.DataImportHandler
  2. org.apache.solr.handler.dataimport.DataImporter
  3. org.apache.solr.handler.dataimport.DocBuilder

Upvotes: 1

Jayesh Bhoyar
Jayesh Bhoyar

Reputation: 727

In first place for full indexing why does your database returns 0 results?

Can you show us what excat query you are firing for full indexing?

I could understand that DB is returning 0 results for partial indexing but for full indexing this should not be a case.

Show us the data config dot xml file that you have written for this issue.

You should fix the issue at database / query side rather than looking for solution in data import handler

Upvotes: 1

Related Questions