CODEWITHSUNDEEP
javamemoryjvm
wFateem
wFateem

Reputation: 346

java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code

I was wondering if some of the JVM gurus out there can briefly explain the following error. What does it actually mean in technical terms and what are the sequences of events that can lead to this error?

java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code

Upvotes: 7

Views: 16676

Answers (3)

z atef
z atef

Reputation: 7679

This can also happen due to disk space issues.

Upvotes: 8

Сергей Кондурушкин
Сергей Кондурушкин

Reputation: 21

This error may be induced by corrupted index which in turn becomes corrupted due to java.io.IOException: No space left on device

Look into your_kafka_home/logs/server.log

It all starts like this:

[2019-11-22 17:27:25,204] FATAL [Replica Manager on Broker 1]: Halting due to unrecoverable I/O error while handling produce request:  (kafka.server.ReplicaManager)
kafka.common.KafkaStorageException: I/O exception in append to log 'xxx-response-1-0'
    at kafka.log.Log.append(Log.scala:443)
    at kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:451)
Caused by: java.io.IOException: No space left on device
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:65)
    at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
    at org.apache.kafka.common.record.MemoryRecords.writeFullyTo(MemoryRecords.java:82)
    at org.apache.kafka.common.record.FileRecords.append(FileRecords.java:159)
    at kafka.log.LogSegment.append(LogSegment.scala:110)
    at kafka.log.Log.append(Log.scala:429)
    ... 21 more

is followed by (possibly mutilple times):

[2019-11-22 17:28:00,395] WARN Found a corrupted index file due to requirement
failed: Corrupt index found, index file (/opt/kafka/kafka-logs/__consumer_offsets- 
9/00000000000000000000.index) has non-zero size but the last offset is 0 which is no
larger than the base offset 0.}. deleting /opt/kafka/kafka-logs/__consumer_offsets-9
/00000000000000000000.timeindex, /opt/kafka/kafka-logs/__consumer_offsets-9
/00000000000000000000.index and rebuilding index... (kafka.log.Log)

and finally comes to :

[2019-11-22 17:28:02,025] FATAL [Kafka Server 1], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.InternalError: **a fault occurred in a recent unsafe memory access operation in compiled Java code**
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
    at kafka.log.TimeIndex.maybeAppend(TimeIndex.scala:107)
    at kafka.log.LogSegment.$anonfun$recover$1(LogSegment.scala:238

Upvotes: 2

apangin
apangin

Reputation: 98324

This error means that sun.misc.Unsafe.getX() or putX() memory access resulted in SIGBUS error, which was then caught by JVM and translated to asynchronous InternalError.

A bit more details:

  • sun.misc.Unsafe is JDK private API that allows to access native memory directly from Java. This API is a foundation for Direct ByteBuffers and particularly MappedByteBuffers.

  • In certain cases an access to memory-mapped region of a file may lead to OS-level exception, namely SIGBUS. Typical examples are:

    1. A memory-mapped buffer is accessed after the underlying file has been truncated.
    2. A file on a network drive has been mapped to memory, and the mapped buffer is accessed after the network connection has been lost.
    3. An attempt to write to a page mapped to a file on tmpfs filesystem results in out-of-memory (by default tmpfs space is limited by 50% of total RAM).

  • HotSpot JVM cannot efficiently detect these problems beforehand. It compiles Unsafe.getX / putX calls to a simple memory access instruction. Additional checks to see if a memory region is valid would be too expensive.

  • Instead JVM handles SIGBUG signal. If it sees the error has happened in Unsafe call, it posts InternalError to current thread and continues execution.
  • IOException would be more appropriate, but JVM cannot throw it or any other exception, since ByteBuffer public contract does not allow its get/put methods to throw any exception.
  • If Unsafe memory access has failed in JIT-compiled method, JVM does not throw an exception immediately (again, it would be too expensive for such hot ByteBuffer API). Instead it posts asynchronous InternalError to the current thread. It means the error would be actually thrown at the nearest native method or at the nearest call to VM runtime. Hence the word "recent" in the error message.

See JDK-4454115 comments describing the solution for the related JDK bug.

Upvotes: 17

Related Questions