CODEWITHSUNDEEP
javascalahadoopmockitopowermock
rahman
rahman

Reputation: 4948

Failed to transform org.apache.hadoop.fs.FileSystem$Cache. org.apache.hadoop.fs.FileSystem$Cache$Key class is frozen

I am trying to mock the hadoop filesystem in my scala test. Any Idea how to go around this please:

import java.net.URI
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem
import org.junit.Test
import org.junit.runner.RunWith
import org.mockito.ArgumentMatchers.any
import org.mockito.Mockito._
import org.powermock.api.mockito.PowerMockito
import org.powermock.core.classloader.annotations.PrepareForTest
import org.powermock.modules.junit4.PowerMockRunner

@RunWith(classOf[PowerMockRunner])
@PrepareForTest(Array(classOf[FileSystem]))  
class SimpleFileSystemTest {

  @Test
  def testMockStaticFileSystem(): Unit = {
    // Mock static method FileSystem.get()
    PowerMockito.mockStatic(classOf[FileSystem])
    val mockFileSystem = mock(classOf[FileSystem])

    // Define behavior of FileSystem.get()
    when(FileSystem.get(any(classOf[URI]), any(classOf[Configuration]))).thenReturn(mockFileSystem)

    // Add simple assertions to verify the behavior
    val uri = new URI("s3a://bucket")
    val conf = new Configuration()

    // Verify that the mocked FileSystem is returned
    val fileSystem = FileSystem.get(uri, conf)
    assert(fileSystem eq mockFileSystem)

    // Verify that the static method was called
    PowerMockito.verifyStatic(classOf[FileSystem], times(1))
    FileSystem.get(uri, conf)
  }
}

but I get this exception:

org.mockito.exceptions.base.MockitoException: 
Mockito cannot mock this class: class org.apache.hadoop.fs.FileSystem.

Mockito can only mock non-private & non-final classes.
If you're not sure why you're getting this error, please report to the mailing list.


Java               : 1.8
JVM vendor name    : Amazon.com Inc.
JVM vendor version : 25.422-b05
JVM name           : OpenJDK 64-Bit Server VM
JVM version        : 1.8.0_422-b05
JVM info           : mixed mode
OS name            : Mac OS X
OS version         : 14.7


Underlying exception : java.lang.IllegalStateException: Failed to transform class with name org.apache.hadoop.fs.FileSystem$Cache. Reason: org.apache.hadoop.fs.FileSystem$Cache$Key class is frozen

    at SimpleFileSystemTest.testMockStaticFileSystem(SimpleFileSystemTest.scala:19)
    at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:68)
    at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:89)
    at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:97)
    at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:87)
    at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:50)
    at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
    at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
    at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
    at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
    at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.lang.IllegalStateException: Failed to transform class with name org.apache.hadoop.fs.FileSystem$Cache. Reason: org.apache.hadoop.fs.FileSystem$Cache$Key class is frozen
    at org.powermock.core.classloader.javassist.JavassistMockClassLoader.defineAndTransformClass(JavassistMockClassLoader.java:119)
    at org.powermock.core.classloader.MockClassLoader.loadMockClass(MockClassLoader.java:174)
    at org.powermock.core.classloader.MockClassLoader.loadClassByThisClassLoader(MockClassLoader.java:102)
    at org.powermock.core.classloader.DeferSupportingClassLoader.loadClass1(DeferSupportingClassLoader.java:147)
    at org.powermock.core.classloader.DeferSupportingClassLoader.loadClass(DeferSupportingClassLoader.java:98)

These are my dependencies:

libraryDependencies ++= Seq(
      "org.slf4j" % "slf4j-api" % "2.0.7" % "provided",
      "com.google.guava" % "guava" % "32.1.2-jre" % "test",
      "org.scalatest" %% "scalatest" % "3.2.16" % "test",
      "org.scalatestplus" %% "mockito-3-4" % "3.2.10.0" % "test",
      "org.powermock" % "powermock-module-junit4" % "2.0.9" % "test",
      "org.powermock" % "powermock-api-mockito2" % "2.0.9" % "test" exclude("org.mockito", "mockito-core"),
      "org.mockito" % "mockito-core" % "3.12.4" % "test",

Upvotes: 0

Views: 58

Answers (1)

stevel
stevel

Reputation: 13480

  1. You are trying to mock FileSystem, and it is abstract. You need to mock a subclass of it like RawLocalFileSystem. Consider that class, stable.

  2. Avoid mocking S3AFS as it's complicated and moves its internals around a lot in ways that even break its own mocked tests. Consider that class's internal methods unstable.

  3. There's a package private static method FileSystem.addFileSystemForTesting() which lets you push any FS subclass into the cache and so have it returned on a get() call. You need to add some class in the package to get at it; hadoop-common test JAR has a class FileSystemTestHelper which provides public access to it from outside the package. Consider this stable.

  4. if you really, really want/need to subclass S3A FS, the hadoop-aws test JAR has a class org.apache.hadoop.fs.s3a.MockS3AFileSystem which is actually a subclass of S3AFS; it relays calls to a (usually mockito) mocked fs where you can tune public methods without worrying about most of the complexity of s3a code changes. This isn't for public use though and has no stability guarantees. If an update breaks your code: your homework to fix that.

Upvotes: 1

Related Questions