CODEWITHSUNDEEP
pythonmysqlsolr
Ajit Kumar
Ajit Kumar

Reputation: 41

How to configure and run Solr full dataimport from MySQL using Python?

I need to perform full import or delta import programmatically using python and mysql. I am aware of the process in java. We can do it in following way:

CommonsHttpSolrServer server = new CommonsHttpSolrServer("http://localhost:8983/solr");
ModifiableSolrParams params = new ModifiableSolrParams();
params.set("command", "full-import");
QueryRequest request = new QueryRequest(params);
request.setPath("/dataimport");
server.request(request);

I am trying to implement it in python. Can you suggest the equivalent code in python or any solr python api that supports this?

Upvotes: 0

Views: 723

Answers (2)

James Doepp - pihentagyu
James Doepp - pihentagyu

Reputation: 1308

There are a few python APIs, but I use mysolr (http://mysolr.readthedocs.io/en/latest/user/userguide.html) because you can use json in indexing, making it faster. 

    from mysolr import Solr

    ## For full index, delete all data after final commit:
    solr.delete_by_query('*:*', commit=False)

    solr = Solr("http://localhost:8983/solr/collection", version=4)
    documents = [
        {'id' : 1,
         'field1' : 'foo'
        },
        {'id' : 2,
         'field1' : 'bar'
        }
    ]

    solr.update(documents, 'json', commit=False)
    solr.commit()

You can query like 1000 records at a time, create a list of them ("documents" above), and send them to the solr index. Then when finished, do the commit. If it's a full query, you can clear all data without committing, and the old data will be deleted once you do the final commit.

Upvotes: 2

MatsLindh
MatsLindh

Reputation: 52832

You trigger the DataImportHandler by making a single HTTP request, and the Java example is just a way to do that using the SolrJ package.

In native python3 you can do this by using urllib.request:

import urllib.request
urllib.request.urlopen('http://localhost:8983/solr/collection/dataimport?command=full-import')

In python2 the same function is available under urllib2:

import urllib2
urllib2.urlopen('http://localhost:8983/solr/collection/dataimport?command=full-import')

Or if you're using the requests library (which can be installed through pip install requests):

import requests
requests.get('http://localhost:8983/solr/collection/dataimport?command=full-import')

Upvotes: 1

Related Questions