Reputation: 703

java.lang.NoSuchMethodError in Flink

I trying to read the a file using :

final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet<String> line = env.readTextFile("file:///pathtofile/myfile.txt");

I get following error:

java.lang.NoSuchMethodError: org.apache.flink.api.common.io.DelimitedInputFormat: method <init>(Lorg/apache/flink/core/fs/Path;)V not found

I'm using flink version 1.3.2, java version "1.8.0_91"

Upvotes: 2

Answers (7)

sahil dorwat

Reputation: 49

I faced the similar issue, for me the problem was flink minor version mismatch. My local flink cluster was running flink-1.8.0 and my code expected the version to be flink-1.8.3. Switching to newer version solved this issue.

Upvotes: 1

mohaseeb

Reputation: 389

Another source of this error could be a mismatch of scala version between Flink and your application code (or your application dependencies that uses scala).

For instance, in my case, I was using Flink 1.7.1 and I had to update my scala dependencies from 2.11 to 2.12; I updated the artifcatId of the concerned dependencies as follows: from flink-scala_2.11 to flink-scala_2.12, flink-table_2.11 to flink-table_2.12, etc.

See here for more info.

Upvotes: 0

Kulasangar

Reputation: 9434

For the ones who're stuggling with the above issue, while using Flink 1.3.2, here's the entire pom which I was able to successfully build:

<build>
      <plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-assembly-plugin</artifactId>
            <version>2.4.1</version>
            <configuration>
               <!-- get all project dependencies -->
               <descriptorRefs>
                  <descriptorRef>jar-with-dependencies</descriptorRef>
               </descriptorRefs>
               <!-- MainClass in mainfest make a executable jar -->
               <archive>
                  <manifest>
                     <mainClass>FlinktoLambda</mainClass>
                  </manifest>
               </archive>
            </configuration>
            <executions>
               <execution>
                  <id>make-assembly</id>
                  <!-- bind to the packaging phase -->
                  <phase>package</phase>
                  <goals>
                     <goal>single</goal>
                  </goals>
               </execution>
            </executions>
         </plugin>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <configuration>
               <archive>
                  <manifest>
                     <mainClass>FlinktoLambda</mainClass>
                  </manifest>
               </archive>
            </configuration>
         </plugin>
         <!--added newly-->
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.1</version>
            <configuration>
               <source>1.8</source>
               <target>1.8</target>
               <compilerId>jdt</compilerId>
            </configuration>
            <dependencies>
               <dependency>
                  <groupId>org.eclipse.tycho</groupId>
                  <artifactId>tycho-compiler-jdt</artifactId>
                  <version>0.21.0</version>
               </dependency>
            </dependencies>
         </plugin>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.1.0</version>
            <executions>
               <execution>
                  <phase>package</phase>
                  <goals>
                     <goal>shade</goal>
                  </goals>
                  <configuration>
                     <transformers>
                        <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                           <mainClass>FlinktoLambda</mainClass>
                        </transformer>
                        <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                           <resource>reference.conf</resource>
                        </transformer>
                     </transformers>
                     <relocations>
                        <relocation>
                           <pattern>org.codehaus.plexus.util</pattern>
                           <shadedPattern>org.shaded.plexus.util</shadedPattern>
                           <excludes>
                              <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
                              <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
                           </excludes>
                        </relocation>
                     </relocations>
                  </configuration>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>
   <name>my-app</name>
   <url>http://maven.apache.org</url>
   <dependencyManagement>
      <dependencies>
         <dependency>
            <groupId>FlinktoLambda</groupId>
            <artifactId>my-app</artifactId>
            <exclusions>
               <exclusion>
                  <groupId>com.typesafe.akka</groupId>
                  <artifactId>akka-remote_2.10</artifactId>
               </exclusion>
               <exclusion>
                  <groupId>com.typesafe.akka</groupId>
                  <artifactId>akka-actor_2.10</artifactId>
               </exclusion>
               <exclusion>
                  <groupId>com.typesafe.akka</groupId>
                  <artifactId>akka-slf4j_2.10</artifactId>
               </exclusion>
            </exclusions>
         </dependency>
      </dependencies>
   </dependencyManagement>
   <dependencies>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-table_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-java</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-streaming-java_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-clients_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-scala_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-streaming-scala_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
   </dependencies>

Please change your main class accordingly in the shade plugin.

Upvotes: 0

Amarjit Dhillon

Reputation: 2816

Are you getting this error in IntelliJ or Dashboard, if you are getting this error in IntelliJ then make sure you use the same Flink version in your pom.xml and also add dependency shading in the build like this

 <build>

        <plugins>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>

                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <compilerId>jdt</compilerId>
                </configuration>

                <dependencies>
                    <dependency>
                        <groupId>org.eclipse.tycho</groupId>
                        <artifactId>tycho-compiler-jdt</artifactId>
                        <version>0.21.0</version>
                    </dependency>
                </dependencies>
            </plugin>


            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.1.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <relocations>
                                <relocation>
                                    <pattern>org.codehaus.plexus.util</pattern>
                                    <shadedPattern>org.shaded.plexus.util</shadedPattern>
                                    <excludes>
                                        <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
                                        <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
                                    </excludes>
                                </relocation>
                            </relocations>
                        </configuration>
                    </execution>
                </executions>
            </plugin>


        </plugins>


</build>>

make sure to run maven clean install in the terminal after you make changes . On the other hand, If you are having this issue only in Dashboard not in intelliJ , then have a look here

Upvotes: 2

gaurav1207

Reputation: 703

One possible cause for error "java.lang.NoSuchMethodError" is when we use different version of flink then what we have installed on our system. For me, I have Flink 1.3.2 and the version I was using was 1.1.1 . So I updated my pom file to have same version.

Upvotes: 1

Yaroslav

Reputation: 466

There is a conflict with dependencies. Apache Flink loads many classes by default into its classpath.

Please read this article https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/debugging_classloading.html the last section

Resolving Dependency Conflicts with Flink using the maven-shade-plugin

Apache Flink loads many classes by default into its classpath. If a user uses a different version of a library that Flink is using, often

IllegalAccessExceptions

NoSuchMethodError

are the result.

So, I suggest to play with your pom.xml and use maven-shade-plugin and add correct relocation, as we have in example

<relocation>
  <pattern>org.codehaus.plexus.util</pattern>
  <shadedPattern>org.shaded.plexus.util</shadedPattern>
  <excludes>
    <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
    <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
  </excludes>
</relocation>

Upvotes: 4

João Pedro Abubakir

Reputation: 56

You need to check your build path, make sure the libs are there and import them properly

Upvotes: 0

java.lang.NoSuchMethodError in Flink

Answers (7)

Related Questions