CODEWITHSUNDEEP
javamavenluceneapache-nifi
scarpacci
scarpacci

Reputation: 9194

Can't override a maven dependency from third party lib -NiFi

I have been working on creating a custom NIFI processor. I am trying to use the luwak search library, which uses Lucene 6.5.0. However, the NiFi libs, seem to use Lucene 4.10.4. I have tried all kinds of things with the pom file, but can't get it to override the NiFi 4.10.4 lib when I build the nar file. Highly likely this is user error. I just can't for the life of me figure out what it is. Runs fine in my IDE, but when I build the artifact (NAR), I can see that the 4.10.4 libs are there, but not some of the 6.5.0 that Luwak needs. Any help would be greatly appreciated.

Reading an research made it seem like if I specifically put the Lucene versions for libs I needed in the pom with DependencyMangement that it would work. Also tried exclusions (also shown).

EDITED: Tried what @Rob suggested and removed pom import. However, still having the same issue. Updated Pom Example

Maven Dependency Tree:

    [INFO] gov.pnnl.nifi:nifi-streamqry-nar:nar:1.3
[INFO] \- gov.pnnl.nifi:nifi-streamqry-processors:jar:1.3:compile
[INFO]    \- com.github.flaxsearch:luwak:jar:1.5.0:compile
[INFO]       +- org.apache.lucene:lucene-core:jar:4.10.4:compile (version managed from 6.5.0)
[INFO]       +- org.apache.lucene:lucene-memory:jar:6.5.0:compile
[INFO]       |  \- (org.apache.lucene:lucene-core:jar:4.10.4:compile - version managed from 6.5.0; omitted for duplicate)
[INFO]       +- org.apache.lucene:lucene-analyzers-common:jar:4.10.4:compile (version managed from 6.5.0)
[INFO]       |  \- (org.apache.lucene:lucene-core:jar:4.10.4:compile - version managed from 6.5.0; omitted for duplicate)
[INFO]       +- org.apache.lucene:lucene-queries:jar:6.5.0:compile
[INFO]       |  \- (org.apache.lucene:lucene-core:jar:4.10.4:compile - version managed from 6.5.0; omitted for duplicate)
[INFO]       \- org.apache.lucene:lucene-queryparser:jar:4.10.4:compile (version managed from 6.5.0)
[INFO]          \- (org.apache.lucene:lucene-core:jar:4.10.4:compile - version managed from 6.5.0; omitted for duplicate)

POM File

<properties>
    <lucene.group>org.apache.lucene</lucene.group>
    <lucene.version>6.5.0</lucene.version>
</properties>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>${lucene.group}</groupId>
            <artifactId>lucene-core</artifactId>
            <version>6.5.0</version>
        </dependency>
        <dependency>
            <groupId>${lucene.group}</groupId>
            <artifactId>lucene-memory</artifactId>
            <version>${lucene.version}</version>
        </dependency>
        <dependency>
            <groupId>${lucene.group}</groupId>
            <artifactId>lucene-queryparser</artifactId>
            <version>${lucene.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.lucene/lucene-queries -->

        <dependency>
            <groupId>${lucene.group}</groupId>
            <artifactId>lucene-analyzers-common</artifactId>
            <version>${lucene.version}</version>
        </dependency>
        <dependency>
            <groupId>${lucene.group}</groupId>
            <artifactId>lucene-queries</artifactId>
            <version>${lucene.version}</version>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.apache.nifi</groupId>
        <artifactId>nifi-api</artifactId>
        <scope>compile</scope>
        <exclusions>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-core</artifactId>
            </exclusion>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-queryparser</artifactId>
            </exclusion>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-analyzers-common</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.nifi</groupId>
        <artifactId>nifi-utils</artifactId>
        <exclusions>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-core</artifactId>
            </exclusion>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-queryparser</artifactId>
            </exclusion>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-analyzers-common</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.nifi</groupId>
        <artifactId>nifi-mock</artifactId>
        <scope>test</scope>
        <exclusions>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-core</artifactId>
            </exclusion>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-queryparser</artifactId>
            </exclusion>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-analyzers-common</artifactId>
            </exclusion>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-sandbox</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

Exclusions:

<dependencies>
    <dependency>
        <groupId>org.apache.nifi</groupId>
        <artifactId>nifi-api</artifactId>
        <scope>compile</scope>
        <exclusions>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-core</artifactId>
            </exclusion>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-queryparser</artifactId>
            </exclusion>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-analyzers-common</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.nifi</groupId>
        <artifactId>nifi-utils</artifactId>
        <exclusions>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-core</artifactId>
            </exclusion>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-queryparser</artifactId>
            </exclusion>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-analyzers-common</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.nifi</groupId>
        <artifactId>nifi-mock</artifactId>
        <scope>test</scope>
        <exclusions>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-core</artifactId>
            </exclusion>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-queryparser</artifactId>
            </exclusion>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-analyzers-common</artifactId>
            </exclusion>
            <exclusion>
                <groupId>${lucene.group}</groupId>
                <artifactId>lucene-sandbox</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>

Upvotes: 0

Views: 737

Answers (2)

Bryan Bende
Bryan Bende

Reputation: 18670

The 4.x lucene versions are not coming in through transitive dependencies so excluding them from things like nifi-api and nifi-mock will not do anything.

The reason you are getting the 4.x version is because your NAR bundle root pom most likely has a parent of nifi-nar-bundles, which means you are inheriting all the way up to NiFi's root pom, which cause's NiFi's dependencyManagement to take effect in your NAR.

You have two options...

1) If this is a custom bundle that is going to live outside the NiFi source tree, then you can remove this relationship between your bundle and NiFi. This is described here:

https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-Inheritance

The latest NAR plugin is 1.2.0.

2) If you were planning to contribute this back to NiFi, or if you are maintaining your own fork of NiFi and want your bundle to live under nifi/nifi-nar-bundles, then you should be able to make this work by declaring your own dependencyManagement section in the root pom of your bundle, and declaring the same lucene dependencies as in the top-level NiFi pom, and setting their version to 6.5.0.

There are some examples of similar situations in NiFi already..

For example, NiFi's top-level pom has a specific version of http-client declared, and the Solr processors need a different version, so it is redeclared here:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-solr-bundle/pom.xml#L43-L45

The important part of this second option is that your dependencyManagement section has to be in the pom of your bundle that sits above your processors and NAR.

Upvotes: 3

Rob
Rob

Reputation: 6497

Your problem is caused by

<type>pom</type>
<scope>import</scope>

From the Maven documentation:

import (only available in Maven 2.0.9 or later) This scope is only supported on a dependency of type pom in the section. It indicates the dependency to be replaced with the effective list of dependencies in the specified POM's section. Since they are replaced, dependencies with a scope of import do not actually participate in limiting the transitivity of a dependency.

You almost certainly want to be working with the jar artifacts. It will probably work as you desire if you simply remove the and tags.

By explicitly mentioning the group/artifact with a specific version, maven will prefer that version over those that are included through transitive dependencies. As long as the group/artifact names have not changed, then this approach works well.

However, if the group/artifact names have changed, you will need to exclude the dependency in the tag that is pulling in the obsolete artifact.

Upvotes: 2

Related Questions