Flink with shaded zookeeper: java.lang.NoSuchMethodError for QuorumMaj

Question

I'm deploying a Flink standaloneApplication mode with high availability enabled (via zookeeper).

Bot the JobManager and the TaskManager initialization fail on:

[2022-09-07 09:09:28,871] ERROR [org.apache.flink.runtime.entrypoint.ClusterEntrypoint] - Could not start cluster entrypoint StandaloneApplicationClusterEntryPoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint StandaloneApplicationClusterEntryPoint.
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:250) ~[jobs-0.1.1-20220907.080009-87.jar:?]
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:711) [jobs-0.1.1-20220907.080009-87.jar:?]
    at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:82) [jobs-0.1.1-20220907.080009-87.jar:?]
Caused by: java.lang.NoSuchMethodError: 'void org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.quorum.flexible.QuorumMaj.<init>(java.util.Map)'
    at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.<init>(EnsembleTracker.java:57) ~[jobs-0.1.1-20220907.080009-87.jar:?]
    at org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.<init>(CuratorFrameworkImpl.java:187) ~[jobs-0.1.1-20220907.080009-87.jar:?]
    at org.apache.flink.shaded.curator5.org.apache.curator.framework.CuratorFrameworkFactory$Builder.build(CuratorFrameworkFactory.java:188) ~[jobs-0.1.1-20220907.080009-87.jar:?]
    at org.apache.flink.runtime.util.ZooKeeperUtils.startCuratorFramework(ZooKeeperUtils.java:289) ~[jobs-0.1.1-20220907.080009-87.jar:?]
    at org.apache.flink.runtime.util.ZooKeeperUtils.startCuratorFramework(ZooKeeperUtils.java:274) ~[jobs-0.1.1-20220907.080009-87.jar:?]
    at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createZooKeeperHaServices(HighAvailabilityServicesUtils.java:90) ~[jobs-0.1.1-20220907.080009-87.jar:?]
    at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:140) ~[jobs-0.1.1-20220907.080009-87.jar:?]
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:427) ~[jobs-0.1.1-20220907.080009-87.jar:?]
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:376) ~[jobs-0.1.1-20220907.080009-87.jar:?]
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:277) ~[jobs-0.1.1-20220907.080009-87.jar:?]
    at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:227) ~[jobs-0.1.1-20220907.080009-87.jar:?]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:?]

I'm building the a fat jar with maven and I have included compile dependencies on shaded-zookeper from flink (version 1.15.2 with Java 11). Pom looks like:

<...

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>${flink.version}</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-core</artifactId>
            <version>${flink.version}</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java</artifactId>
            <version>${flink.version}</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-avro</artifactId>
            <version>${flink.version}</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-container</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-runtime</artifactId>
            <version>${flink.version}</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-metrics-core</artifactId>
            <version>${flink.version}</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-shaded-zookeeper-3</artifactId>
            <version>3.6.3-15.0</version>
            <scope>compile</scope>
        </dependency>

../>

I tried adding the shaded curator in the pom file (and explicitly in classpath) but also didn't work. I have also tried using shaded zookeeper version 3.4 and 3.5 but also without any success.

Is there an incompatibility with zookeeper and curator? Any ideas how to proceed?

Answer 1

My guess is that you somehow end up with flink-shaded-zookeeper 3.4 on the classpath. Please check the logs for the classpath (near the top) and scan for additional flink-shaded-zookeeper jars.