CODEWITHSUNDEEP
solrdataimporthandler
Adam
Adam

Reputation: 6172

Getting Data-Import-Handler to work with Solr 8.10.1

I'm using regular Solr 8.10.1 (no Solr Cloud) I created a new core by cd into solr bin directory and ran solr create -c "mytest"

I start it like C:\solr-8.10.1\bin\solr start -p 8983

My folder structure:

solrconfig.xml

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
  <lst name="defaults">
    <str name="config">sample-data-config.xml</str>
  </lst>
</requestHandler>
    
<lib dir="./lib" />

I also tried this, but I get the same error

<lib dir="./solr/lib" />

I copied these files into the server\lib folder: solr-dataimporthandler-8.10.1.jar and solr-dataimporthandler-extras-8.10.1.jar.

- solr-8.10.1
    - server
        - lib
        solr-dataimporthandler-8.10.1.jar
        solr-dataimporthandler-extras-8.10.1.jar
        - solr
            - configsets
                - sample_techproducts_configs
                    - conf
            - mytest
                - conf
                    - lang
                  data-config.xml
                  managed-schema
                  protwords.txt
                  solrconfig.xml A
                  stopwords.txt
                  synonyms.txt
                - data
        solr.xml

sample-data-config

<dataConfig>
<dataSource driver="com.microsoft.sqlserver.jdbc.SQLServerDriver" url="jdbc:sqlserver://localhost:1433;databaseName=test" user="<username>" password="<pwd>" />
    <document name="catalogitems">
        <entity pk="id" name="catalogitem" query="SELECT id,photo from products">
            <field name="id" column="ID" />
            <field name="photo" column="photo" />
        </entity>
    </document>
</dataConfig>

When I run http://localhost:8983/solr/mytest/dataimport?command=full-import

I get error:

HTTP ERROR 500 java.lang.NoClassDefFoundError: org/apache/solr/util/plugin/SolrCoreAware
URI:    /solr/mytest/dataimport
STATUS: 500
MESSAGE:    java.lang.NoClassDefFoundError: org/apache/solr/util/plugin/SolrCoreAware
SERVLET:    default
CAUSED BY:  java.lang.NoClassDefFoundError: org/apache/solr/util/plugin/SolrCoreAware
CAUSED BY:  java.lang.ClassNotFoundException: org.apache.solr.util.plugin.SolrCoreAware
Caused by:
java.lang.NoClassDefFoundError: org/apache/solr/util/plugin/SolrCoreAware
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(Unknown Source)
    at java.security.SecureClassLoader.defineClass(Unknown Source)
    at java.net.URLClassLoader.defineClass(Unknown Source)
    at java.net.URLClassLoader.access$100(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:538)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Unknown Source)
    at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:538)
    at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:483)
    at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:840)
    at org.apache.solr.core.PluginBag.createPlugin(PluginBag.java:154)
    at org.apache.solr.core.PluginBag.init(PluginBag.java:300)
    at org.apache.solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java:130)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1015)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:921)
    at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1448)
    at org.apache.solr.core.CoreContainer.lambda$load$11(CoreContainer.java:869)
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: org.apache.solr.util.plugin.SolrCoreAware
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    ... 35 more

I also checked the error log via the admin console, where I see:

org.apache.solr.core.SolrCoreInitializationException: SolrCore 'mytest' is not available due to init failure: org/apache/solr/util/plugin/SolrCoreAware
    at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1964)
    at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1937)
    at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)
    at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:525)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:427)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:357)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
    at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:516)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.solr.common.SolrException: org/apache/solr/util/plugin/SolrCoreAware
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1086)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:921)
    at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1448)
    at org.apache.solr.core.CoreContainer.lambda$load$11(CoreContainer.java:869)
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ... 1 more
Caused by: java.lang.NoClassDefFoundError: org/apache/solr/util/plugin/SolrCoreAware
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(Unknown Source)
    at java.security.SecureClassLoader.defineClass(Unknown Source)
    at java.net.URLClassLoader.defineClass(Unknown Source)
    at java.net.URLClassLoader.access$100(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:538)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Unknown Source)
    at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:538)
    at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:483)
    at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:840)
    at org.apache.solr.core.PluginBag.createPlugin(PluginBag.java:154)
    at org.apache.solr.core.PluginBag.init(PluginBag.java:300)
    at org.apache.solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java:130)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1015)
    ... 9 more
Caused by: java.lang.ClassNotFoundException: org.apache.solr.util.plugin.SolrCoreAware
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    ... 35 more

I already checked here, but I think my lib folder structure is correct?

UPDATE 1

When I have in solr.config: <lib dir="./lib" /> my admin UI loads.

Based on Eric's comments I checked https://solr.apache.org/guide/8_10/libs.html#lib-directories so I replaced the above part in solr.config with

<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />

I also tried:

<lib dir="./lib" />
<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />

In both latter 2 configurations I still get error HTTP ERROR 500 java.lang.NoClassDefFoundError: org/apache/solr/util/plugin/SolrCoreAware

UPDATE 2

I've now placed solr-dataimporthandler-8.10.1.jar and solr-dataimporthandler-extras-8.10.1.jar in multiple folders:

  1. C:\Databases\solr-8.10.1\dist
  2. C:\Databases\solr-8.10.1\server\lib
  3. C:\Databases\solr-8.10.1\server\solr\mytest\lib

In my C:\Databases\solr-8.10.1\server\solr\mytest\conf\solrconfig.xml

I tried:

But I keep getting the same error.

Also, I need this jar for multiple cores, so I think I should use folder 1 or 2 above?

Folder structure

enter image description here

UPDATE 3

I have not made any changes to any files/configurations, I just unzipped the download and copied the dataimporthandler jars to some more folders (see my update 2 above).

I see no file solrcore.properties anywhere in my Solr directory (can't find anything on Google either) so I'm not sure where to edit that.

On http://localhost:8983/solr I see:

-DSTOP.KEY=solrrocks
-DSTOP.PORT=7983
-Djava.io.tmpdir=C:\Databases\solr-8.10.1\server\tmp
-Djetty.home=C:\Databases\solr-8.10.1\server
-Djetty.host=0.0.0.0
-Djetty.port=8983
-Dlog4j.configurationFile=C:\Databases\solr-8.10.1\server\resources\log4j2.xml
-Dsolr.default.confdir=C:\Databases\solr-8.10.1\server\solr\configsets\_default\conf
-Dsolr.install.dir=C:\Databases\solr-8.10.1
-Dsolr.jetty.inetaccess.excludes=
-Dsolr.jetty.inetaccess.includes=
-Dsolr.log.dir=C:\Databases\solr-8.10.1\server\logs
-Dsolr.log.muteconsole
-Dsolr.solr.home=C:\Databases\solr-8.10.1\server\solr
-Duser.timezone=UTC
-XX:+AlwaysPreTouch
-XX:+ExplicitGCInvokesConcurrent
-XX:+ParallelRefProcEnabled
-XX:+PerfDisableSharedMem
-XX:+UseG1GC
-XX:+UseLargePages
-XX:-OmitStackTraceInFastThrow
-XX:MaxGCPauseMillis=250
-Xms512m
-Xmx512m
-Xss256k

C:\Databases\solr-8.10.1\server\solr\mytest\core.properties

#Written by CorePropertiesLocator
#Tue Nov 02 14:19:06 UTC 2021
name=mytest

C:\Databases\solr-8.10.1\server\solr\mytest\conf\solrconfig.xml

<lib dir="${solr.install.dir:../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />

But still the same error.

Upvotes: 0

Views: 2332

Answers (1)

Charaf
Charaf

Reputation: 324

As of August 2022.

DIH team stopped support at Solr 8.9. 8.10 is therefore not supported (yet?).

Link to the repository: https://github.com/rohitbemax/dataimporthandler

Link to a discussion about supporting it in Solr 9: https://github.com/rohitbemax/dataimporthandler/issues/32

Link to a PR to support v9: https://github.com/rohitbemax/dataimporthandler/pull/33

With these known issues:

  • Only MariaDB connector supported right now. The connectors to be used need to be shipped as part of this package, and currently only MariaDB connector is shipped. (No SQLite for instance).
  • Solr Admin UI's Dataimport tab could be a bit glitchy when used with this package

Upvotes: 0

Related Questions