Connecting solr with aws RDS Mysql through data import handler

Question

I recently started implementing solr-cloud on AWS EC2 for search applications. I have created 2 AWS Ec2 instances with the following configurations ---

1. EC2 Type - t2.medium
2. ram - 4GB
3. Disk Space - 8GB
4. OS - ubuntu 18.04

For the 2 EC2 instances, I have created a security group which allows all inbound traffic. NACL has default settings that allows all inbound traffic as well.

Steps Followed to install Apache Solr -

1. ssh into ec2 :

ssh -i "pem_file" ubuntu@ec2-public-ipv4-address

2. cd to /opt directory
3. run --> sudo apt-update
4. run --> sudo apt-get openjdk-11
5. Check java -version
6. run --> wget https://archive.apache.org/dist/lucene/solr/8.3.0/solr-8.3.0.tgz
7. run --> tar -xvzf solr-8.3.0.tgz
8. export SOLR_HOME=/opt/solr-8.3.0
9. Add /opt/solr-8.3.0 to Path environment variable
10. Update the sudo vim /etc/hosts file with the hosts -- a. public-ip-v4-address-of-ec2 solr-node-1
11. Started Solr using the following command --> sudo bin/solr start -c -p 8983 -h solr-node-1 -force
12. Checked the opened ports using --> sudo lsof -i -P -n | grep LISTEN
13. Created collections, shards and replicas using ---> bin/solr create -c travasko -d sample_techproducts_configs -n travasko_configs -shards 2 -rf 2 -p 8983

I repeated the same process on the other EC2 machine and ran solr on it. Now, to use the data import handler in solr, I edited the following files:

1. solrconfig.xml

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
    <lst name="defaults">
      <str name="config">data-config.xml</str>
    </lst>
</requestHandler>

2. data-config.xml

<dataConfig>
<dataSource type="JdbcDataSource" 
            driver="com.mysql.jdbc.Driver"
            url="jdbc:mysql://examplerds.cuhj86yfdpid.us-east-1.rds.amazonaws.com:3306/TRAVASKODB1" 
            user="examplerds" 
            password="examplerds#123"/>
<document>
  <entity name="MOMENTS"  
    pk="MOMENT_ID"
    query="SELECT MOMENT_ID,MOMENT_TEXT FROM MOMENTS"
    deltaImportQuery="SELECT MOMENT_ID,MOMENT_TEXT FROM MOMENTS WHERE MOMENT_ID='${dih.delta.MOMENT_ID}'"
    deltaQuery="SELECT MOMENT_ID FROM MOMENTS  WHERE LAST_MODIFIED > '${dih.last_index_time}'"
    >
     <field column="MOMENT_ID" name="MOMENT_ID"/>
     <field column="MOMENT_TEXT" name="MOMENT_TEXT"/>       
  </entity>
</document>
</dataConfig>

3. managed_schema

<schema name="MOMENTS" version="1.5">
    <field name="_version_" type="long" indexed="true" stored="true"/>
    <field name="MOMENT_ID" type="integer" indexed="true" stored="true" required="true" multiValued="false" /> 
    <field name="MOMENT_TEXT" type="string" indexed="true" stored="true" multiValued="false" />
</schema>

4. Downloaded mysql jdbc using the following command:

wget -q "http://search.maven.org/remotecontent?filepath=mysql/mysql-connector-java/5.1.32/mysql-connector-java-5.1.32.jar" -O mysql-connector-java.jar

5. Add to solrconfig.xml:

<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />

<lib dir="${solr.install.dir:../../../..}/dist/" regex="mysql-connector-java.jar" />

6. After editing the files above, I uploaded them to the solr-cloud using the following zookeper command -->

bin/solr zk -n travasko_config -z solr-node-1:9983 cp /opt/solr-8.3.0/server/solr/configsets/_default/conf/managed-schema zk:/configs/travasko_config/managed-schema

7. I then checked all the above files in the solr-cloud and could notice the changes i added.
8. The current issue is that when I select the collection I created above, and click on Dataimport, It throws an error as below --->

The solrconfig.xml file for this index does not have an operational DataImportHandler defined!

Note: The AWS RDS and EC2 instances are in the same VPC sharing the same Security Group.

So why is solrconfig.xml file throwing an error during dataimport ? What am i missing here?

Answer 1

The solution to the above issue was basically setting the java system property for solr versions greater than 8.2.0 as below:

-Denable.dih.dataConfigParam=true

This parameter can be set either in solr.in.cmd or solr.in.sh which can be found inside the directory below: ,

/opt/solr-8.3.0/bin 

If, /opt/solr-8.3.0 is the installation directory of solr.

The other method was to pass this parameter as command line parameter while starting solr as below:

sudo bin/solr start -c -p 8983 -h solr-node-1 -Denable.dih.dataConfigParam=true -force

solr-node-1 is the public IPv4 address of the AWS Ec2 instance on which solr is configured.