Java Native Method to convert Long Array to Byte array

Question

is there any native method in java to copy/convert long array to byte array and vice versa. I am aware of below method

ByteBuffer bb = ByteBuffer.allocate(longArray.length * Long.BYTES);
bb.asLongBuffer().put(longArray);
return bb.array();

but the above method is very very slow especially when our Java application is dealing with large amount is data.

System.arraycopy is a wonderful performance of copying array of same type. if java claim that system.arraycopy is using a native c method, then why they don't include to copy array of long/int to array of bytes as in C memcpy do this job.

appreciate any help.

thank you.

Answer 1

ByteBuffer.asLongBuffer().put() is the right way to copy between different array types. It's simple, pure Java and it's not that slow. I'll demonstrate below.

Note that to make the results equivalent to memcpy, you need to switch ByteBuffer to the native byte order. By default, ByteBuffers are BIG_ENDIAN, while x86 architecture is LITTLE_ENDIAN. Switching to the native byte order will also make copying faster.

bb.order(ByteOrder.nativeOrder());

There are a few other ways to convert arrays worth mentioning.

1. Unsafe.copyMemory()
2. JNI GetPrimitiveArrayCritical + SetByteArrayRegion.

sun.misc.Unsafe is a JDK private, unsupported and deprecated API, but it still works in all versions, at least from JDK 6 to JDK 14. The good thing about it is that it's Java API - there's no need to make a native library.

On the contrary, JNI functions require loading a native library, but these functions are standard and supported. The combination of GetPrimitiveArrayCritical + SetByteArrayRegion allows to copy data directly from one array to another without intermediate storage.

HotSpot JVM also has an undocumented extension - Critical Natives, which allows to access Java primitive arrays directly from native code without JNI overhead. But keep in mind that relying on undocumented API makes your code not portable. The the good news is that Critical Natives are compabitlbe with regular native methods, i.e. when you implement both, you can be sure that the code will work everywhere.

What about performance?

I created a JMH benchmark to compare all the discussed techniques.

package bench;

import org.openjdk.jmh.annotations.*;
import sun.misc.Unsafe;

import java.lang.reflect.Field;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;

@State(Scope.Benchmark)
public class LongArrayCopy {

    @Param({"100", "1000", "10000"})
    private int size;

    private long[] longArray;

    @Setup
    public void setup() {
        longArray = new long[size];
    }

    @Benchmark
    public byte[] byteBuffer() {
        ByteBuffer bb = ByteBuffer.allocate(longArray.length * Long.BYTES);
        bb.order(ByteOrder.nativeOrder());
        bb.asLongBuffer().put(longArray);
        return bb.array();
    }

    @Benchmark
    public byte[] jni() {
        byte[] byteArray = new byte[longArray.length * Long.BYTES];
        copy(longArray, byteArray, byteArray.length);
        return byteArray;
    }

    @Benchmark
    public byte[] jniCritical() {
        byte[] byteArray = new byte[longArray.length * Long.BYTES];
        copyCritical(longArray, byteArray, byteArray.length);
        return byteArray;
    }

    @Benchmark
    public byte[] unsafe() {
        byte[] byteArray = new byte[longArray.length * Long.BYTES];
        theUnsafe.copyMemory(longArray, Unsafe.ARRAY_LONG_BASE_OFFSET,
                byteArray, Unsafe.ARRAY_BYTE_BASE_OFFSET,
                byteArray.length);
        return byteArray;
    }

    private static native void copy(long[] src, byte[] dst, int size);

    private static native void copyCritical(long[] src, byte[] dst, int size);

    private static final Unsafe theUnsafe;

    static {
        try {
            Field f = Unsafe.class.getDeclaredField("theUnsafe");
            f.setAccessible(true);
            theUnsafe = (Unsafe) f.get(null);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }

        System.loadLibrary("arraycopy");
    }
}

arraycopy.c

#include <jni.h>
#include <string.h>

JNIEXPORT void Java_bench_LongArrayCopy_copy(JNIEnv* env, jobject unused,
                                             jlongArray src, jbyteArray dst, jint size) {
    void* data = (*env)->GetPrimitiveArrayCritical(env, src, NULL);
    (*env)->SetByteArrayRegion(env, dst, 0, size, (jbyte*)data);
    (*env)->ReleasePrimitiveArrayCritical(env, src, data, JNI_COMMIT);
}

JNIEXPORT void Java_bench_LongArrayCopy_copyCritical(JNIEnv* env, jobject unused,
                                                     jlongArray src, jbyteArray dst,
                                                     jint size) {
    Java_bench_LongArrayCopy_copy(env, unused, src, dst, size);
}

JNIEXPORT void JavaCritical_bench_LongArrayCopy_copyCritical(jint srclen, jlong* src,
                                                             jint dstlen, jbyte* dst,
                                                             jint size) {
    memcpy(dst, src, size);
}

The results on JDK 8u221 (nanoseconds per copying an array of 1000 longs):

Benchmark                  (size)  Mode  Cnt     Score    Error  Units
LongArrayCopy.byteBuffer     1000  avgt   10  3204,239 ± 49,300  ns/op
LongArrayCopy.jni            1000  avgt   10   774,466 ±  2,973  ns/op
LongArrayCopy.jniCritical    1000  avgt   10   545,801 ±  3,643  ns/op
LongArrayCopy.unsafe         1000  avgt   10   552,265 ±  4,212  ns/op

It may look that ByteBuffer is a way slower comparing to other methods. However, ByteBuffer performance has been heavily optimized since JDK 9. If we run the same example on a modern JDK (11 or 14), we'll see that ByteBuffer is actually the fastest way!

JDK 14.0.1

Benchmark                  (size)  Mode  Cnt    Score   Error  Units
LongArrayCopy.byteBuffer     1000  avgt   10  566,038 ± 1,010  ns/op
LongArrayCopy.jni            1000  avgt   10  659,575 ± 2,145  ns/op
LongArrayCopy.jniCritical    1000  avgt   10  575,381 ± 2,283  ns/op
LongArrayCopy.unsafe         1000  avgt   10  602,838 ± 4,587  ns/op

How can ByteBuffer be faster than Unsafe? The trick is that the JVM compiler can vectorize, unroll and inline the copying loop of a ByteBuffer, while Unsafe.copyMemory always calls to JVM runtime.