NiFi: Unable to add external library to ExecuteScript Processor for Python

Question

I want to use Python DB client for Impala in my NiFi jython ExecuteScript Processor.

I've installed required module with pip install impyla and specify path to it in Module Directory field in ExecuteScript Processor.

After that I run script and got an error javax.script.ScriptException: ImportError: No module named _bitarray

With pip show bitarray I found this module location and add it to the Module Directory field (use comma as a separator) and restart NiFi just in case.

But it didn't help. And I've got the same error.

After a little research I figured out that bitarray is actually a C extension.

And _bitarray appears in the next line of bitarray/__init__.py:

from bitarray._bitarray import _bitarray, bitdiff, bits2bytes, _sysinfo

In its turn _bitarray is stored inside _bitarray.so file in the same directory I specified in Module Directory.

Can someone help me how to load bitarray module?

Full stacktrace:

2017-07-12 15:00:57,627 ERROR [Timer-Driven Process Thread-5] o.a.nifi.processors.script.ExecuteScript org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: ImportError: No module named _bitarray in at line number 1 at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:214) ~[nifi-scripting-processors-1.1.2.jar:1.1.2] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.2.jar:1.1.2] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.2.jar:1.1.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_131] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_131] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131] Caused by: javax.script.ScriptException: ImportError: No module named _bitarray in at line number 1 at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:202) ~[jython-standalone-2.7.0.jar:na] at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:42) ~[jython-standalone-2.7.0.jar:na] at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31) ~[jython-standalone-2.7.0.jar:na] at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264) ~[na:1.8.0_131] at org.apache.nifi.processors.script.impl.JythonScriptEngineConfigurator.eval(JythonScriptEngineConfigurator.java:59) ~[nifi-scripting-processors-1.1.2.jar:1.1.2] at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:204) ~[nifi-scripting-processors-1.1.2.jar:1.1.2] ... 11 common frames omitted Caused by: org.python.core.PyException: null at org.python.core.Py.ImportError(Py.java:328) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.import_logic(imp.java:912) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.import_module_level(imp.java:978) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.importName(imp.java:1062) ~[jython-standalone-2.7.0.jar:na] at org.python.core.ImportFunction.call(builtin.java:1280) ~[jython-standalone-2.7.0.jar:na] at org.python.core.PyObject.call(PyObject.java:431) ~[jython-standalone-2.7.0.jar:na] at org.python.core.builtin.import(builtin.java:1232) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.importFromAs(imp.java:1156) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.importFrom(imp.java:1132) ~[jython-standalone-2.7.0.jar:na] at bitarray$py.f$0(/usr/local/lib/python2.7/dist-packages/bitarray/init.py:131) ~[na:na] at bitarray$py.call_function(/usr/local/lib/python2.7/dist-packages/bitarray/init.py) ~[na:na] at org.python.core.PyTableCode.call(PyTableCode.java:167) ~[jython-standalone-2.7.0.jar:na] at org.python.core.PyCode.call(PyCode.java:18) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.createFromCode(imp.java:436) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.createFromPyClass(imp.java:236) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.createFromPyClass(imp.java:205) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.loadFromSource(imp.java:651) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.find_module(imp.java:543) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.import_next(imp.java:840) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.import_module_level(imp.java:959) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.importName(imp.java:1062) ~[jython-standalone-2.7.0.jar:na] at org.python.core.ImportFunction.call(builtin.java:1280) ~[jython-standalone-2.7.0.jar:na] at org.python.core.PyObject.call(PyObject.java:431) ~[jython-standalone-2.7.0.jar:na] at org.python.core.builtin.import(builtin.java:1232) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.importFromAs(imp.java:1156) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.importFrom(imp.java:1132) ~[jython-standalone-2.7.0.jar:na] at impala.hiveserver2$py.f$0(/usr/local/lib/python2.7/dist-packages/impala/hiveserver2.py:1211) ~[na:na] at impala.hiveserver2$py.call_function(/usr/local/lib/python2.7/dist-packages/impala/hiveserver2.py) ~[na:na] at org.python.core.PyTableCode.call(PyTableCode.java:167) ~[jython-standalone-2.7.0.jar:na] at org.python.core.PyCode.call(PyCode.java:18) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.createFromCode(imp.java:436) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.createFromPyClass(imp.java:236) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.createFromPyClass(imp.java:205) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.loadFromSource(imp.java:651) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.find_module(imp.java:543) ~[jython-standalone-2.7.0.jar:na] at org.python.core.PyModule.impAttr(PyModule.java:106) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.import_next(imp.java:842) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.import_logic(imp.java:904) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.import_module_level(imp.java:978) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.importName(imp.java:1062) ~[jython-standalone-2.7.0.jar:na] at org.python.core.ImportFunction.call(builtin.java:1280) ~[jython-standalone-2.7.0.jar:na] at org.python.core.PyObject.call(PyObject.java:431) ~[jython-standalone-2.7.0.jar:na] at org.python.core.builtin.import(builtin.java:1232) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.importOneAs(imp.java:1099) ~[jython-standalone-2.7.0.jar:na] at impala.dbapi$py.f$0(/usr/local/lib/python2.7/dist-packages/impala/dbapi.py:193) ~[na:na] at impala.dbapi$py.call_function(/usr/local/lib/python2.7/dist-packages/impala/dbapi.py) ~[na:na] at org.python.core.PyTableCode.call(PyTableCode.java:167) ~[jython-standalone-2.7.0.jar:na] at org.python.core.PyCode.call(PyCode.java:18) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.createFromCode(imp.java:436) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.createFromPyClass(imp.java:236) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.createFromPyClass(imp.java:205) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.loadFromSource(imp.java:651) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.find_module(imp.java:543) ~[jython-standalone-2.7.0.jar:na] at org.python.core.PyModule.impAttr(PyModule.java:106) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.import_next(imp.java:842) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.import_logic(imp.java:904) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.import_module_level(imp.java:978) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.importName(imp.java:1062) ~[jython-standalone-2.7.0.jar:na] at org.python.core.ImportFunction.call(builtin.java:1280) ~[jython-standalone-2.7.0.jar:na] at org.python.core.PyObject.call(PyObject.java:431) ~[jython-standalone-2.7.0.jar:na] at org.python.core.builtin.import(builtin.java:1232) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.importFromAs(imp.java:1156) ~[jython-standalone-2.7.0.jar:na] at org.python.core.imp.importFrom(imp.java:1132) ~[jython-standalone-2.7.0.jar:na] at org.python.pycode._pyx1450.f$0(:5) ~[na:na] at org.python.pycode._pyx1450.call_function() ~[na:na] at org.python.core.PyTableCode.call(PyTableCode.java:167) ~[jython-standalone-2.7.0.jar:na] at org.python.core.PyCode.call(PyCode.java:18) ~[jython-standalone-2.7.0.jar:na] at org.python.core.Py.runCode(Py.java:1386) ~[jython-standalone-2.7.0.jar:na] at org.python.core.builtin.eval(builtin.java:497) ~[jython-standalone-2.7.0.jar:na] at org.python.core.builtin.eval(builtin.java:501) ~[jython-standalone-2.7.0.jar:na] at org.python.util.PythonInterpreter.eval(PythonInterpreter.java:259) ~[jython-standalone-2.7.0.jar:na] at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:40) ~[jython-standalone-2.7.0.jar:na]

Answer 1

Jython can't load native (CPython, e.g.) modules, please see this related SO answer for more details.