Reputation: 247
is there any native method in java to copy/convert long array to byte array and vice versa. I am aware of below method
ByteBuffer bb = ByteBuffer.allocate(longArray.length * Long.BYTES);
bb.asLongBuffer().put(longArray);
return bb.array();
but the above method is very very slow especially when our Java application is dealing with large amount is data.
System.arraycopy is a wonderful performance of copying array of same type. if java claim that system.arraycopy is using a native c method, then why they don't include to copy array of long/int to array of bytes as in C memcpy do this job.
appreciate any help.
thank you.
Upvotes: 2
Views: 1953
Reputation: 98640
ByteBuffer.asLongBuffer().put()
is the right way to copy between different array types. It's simple, pure Java and it's not that slow. I'll demonstrate below.
Note that to make the results equivalent to memcpy
, you need to switch ByteBuffer
to the native byte order. By default, ByteBuffers are BIG_ENDIAN, while x86 architecture is LITTLE_ENDIAN. Switching to the native byte order will also make copying faster.
bb.order(ByteOrder.nativeOrder());
There are a few other ways to convert arrays worth mentioning.
Unsafe.copyMemory()
GetPrimitiveArrayCritical
+ SetByteArrayRegion
.sun.misc.Unsafe
is a JDK private, unsupported and deprecated API, but it still works in all versions, at least from JDK 6 to JDK 14. The good thing about it is that it's Java API - there's no need to make a native library.
On the contrary, JNI functions require loading a native library, but these functions are standard and supported. The combination of GetPrimitiveArrayCritical
+ SetByteArrayRegion
allows to copy data directly from one array to another without intermediate storage.
HotSpot JVM also has an undocumented extension - Critical Natives, which allows to access Java primitive arrays directly from native code without JNI overhead. But keep in mind that relying on undocumented API makes your code not portable. The the good news is that Critical Natives are compabitlbe with regular native methods, i.e. when you implement both, you can be sure that the code will work everywhere.
I created a JMH benchmark to compare all the discussed techniques.
package bench;
import org.openjdk.jmh.annotations.*;
import sun.misc.Unsafe;
import java.lang.reflect.Field;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
@State(Scope.Benchmark)
public class LongArrayCopy {
@Param({"100", "1000", "10000"})
private int size;
private long[] longArray;
@Setup
public void setup() {
longArray = new long[size];
}
@Benchmark
public byte[] byteBuffer() {
ByteBuffer bb = ByteBuffer.allocate(longArray.length * Long.BYTES);
bb.order(ByteOrder.nativeOrder());
bb.asLongBuffer().put(longArray);
return bb.array();
}
@Benchmark
public byte[] jni() {
byte[] byteArray = new byte[longArray.length * Long.BYTES];
copy(longArray, byteArray, byteArray.length);
return byteArray;
}
@Benchmark
public byte[] jniCritical() {
byte[] byteArray = new byte[longArray.length * Long.BYTES];
copyCritical(longArray, byteArray, byteArray.length);
return byteArray;
}
@Benchmark
public byte[] unsafe() {
byte[] byteArray = new byte[longArray.length * Long.BYTES];
theUnsafe.copyMemory(longArray, Unsafe.ARRAY_LONG_BASE_OFFSET,
byteArray, Unsafe.ARRAY_BYTE_BASE_OFFSET,
byteArray.length);
return byteArray;
}
private static native void copy(long[] src, byte[] dst, int size);
private static native void copyCritical(long[] src, byte[] dst, int size);
private static final Unsafe theUnsafe;
static {
try {
Field f = Unsafe.class.getDeclaredField("theUnsafe");
f.setAccessible(true);
theUnsafe = (Unsafe) f.get(null);
} catch (Exception e) {
throw new RuntimeException(e);
}
System.loadLibrary("arraycopy");
}
}
arraycopy.c
#include <jni.h>
#include <string.h>
JNIEXPORT void Java_bench_LongArrayCopy_copy(JNIEnv* env, jobject unused,
jlongArray src, jbyteArray dst, jint size) {
void* data = (*env)->GetPrimitiveArrayCritical(env, src, NULL);
(*env)->SetByteArrayRegion(env, dst, 0, size, (jbyte*)data);
(*env)->ReleasePrimitiveArrayCritical(env, src, data, JNI_COMMIT);
}
JNIEXPORT void Java_bench_LongArrayCopy_copyCritical(JNIEnv* env, jobject unused,
jlongArray src, jbyteArray dst,
jint size) {
Java_bench_LongArrayCopy_copy(env, unused, src, dst, size);
}
JNIEXPORT void JavaCritical_bench_LongArrayCopy_copyCritical(jint srclen, jlong* src,
jint dstlen, jbyte* dst,
jint size) {
memcpy(dst, src, size);
}
The results on JDK 8u221 (nanoseconds per copying an array of 1000 longs):
Benchmark (size) Mode Cnt Score Error Units
LongArrayCopy.byteBuffer 1000 avgt 10 3204,239 ± 49,300 ns/op
LongArrayCopy.jni 1000 avgt 10 774,466 ± 2,973 ns/op
LongArrayCopy.jniCritical 1000 avgt 10 545,801 ± 3,643 ns/op
LongArrayCopy.unsafe 1000 avgt 10 552,265 ± 4,212 ns/op
It may look that ByteBuffer is a way slower comparing to other methods. However, ByteBuffer performance has been heavily optimized since JDK 9. If we run the same example on a modern JDK (11 or 14), we'll see that ByteBuffer is actually the fastest way!
JDK 14.0.1
Benchmark (size) Mode Cnt Score Error Units
LongArrayCopy.byteBuffer 1000 avgt 10 566,038 ± 1,010 ns/op
LongArrayCopy.jni 1000 avgt 10 659,575 ± 2,145 ns/op
LongArrayCopy.jniCritical 1000 avgt 10 575,381 ± 2,283 ns/op
LongArrayCopy.unsafe 1000 avgt 10 602,838 ± 4,587 ns/op
How can ByteBuffer be faster than Unsafe? The trick is that the JVM compiler can vectorize, unroll and inline the copying loop of a ByteBuffer, while Unsafe.copyMemory
always calls to JVM runtime.
Upvotes: 4