CODEWITHSUNDEEP
javaamazon-s3java-native-interfacejclouds
Bhavana Shah
Bhavana Shah

Reputation: 13

How to upload Direct Buffer gotten from JNI to S3 directly

I have created a shared memory (of size 200MB) which is mapped to both a Java process as well as a C++ process running on the system. The C++ process writes 50MB of data in this shared memory. At the JAVA side, a JNI function which has mapped the same shared memory reads this data into a direct buffer like this:

JNIEXPORT jobject JNICALL Java_service_SharedMemoryJNIService_getDirectByteBuffer
  (JNIEnv *env, jclass jobject, jlong buf_addr, jint buf_len){

        return env->NewDirectByteBuffer((void *)buf_addr, buf_len);

  }

Now, at the JAVA side, I need to upload this 50MB of data to S3. Currently, I have to copy this direct buffer to a buffer in JVM heap like this:

public String uploadByteBuffer(String container, String objectKey, ByteBuffer bb) {

            BlobStoreContext context = getBlobStoreContext();
            BlobStore blobStore = context.getBlobStore();
            byte[] buf = new byte[bb.capacity()];
            bb.get(buf);
            ByteArrayPayload payload = new ByteArrayPayload(buf);
            Blob blob = blobStore.blobBuilder(objectKey)
                    .payload(payload)
                    .contentLength(bb.capacity())
                    .build();
            blobStore.putBlob(container, blob);
            return objectKey;
}

I want to avoid this extra copy form shared memory to JVM heap. Is there a way to directly upload data contained in Direct buffer to S3 ?

Thanks

Upvotes: 0

Views: 73

Answers (1)

Andrew Gaul
Andrew Gaul

Reputation: 2402

BlobBuilder.payload can take a ByteSource and you can use a ByteBuffer wrapper:

public class ByteBufferByteSource extends ByteSource {
   private final ByteBuffer buffer;

   public ByteBufferByteSource(ByteBuffer buffer) {
      this.buffer = checkNotNull(buffer);
   }

   @Override
   public InputStream openStream() {
      return new ByteBufferInputStream(buffer);
   }

   private static final class ByteBufferInputStream extends InputStream {
      private final ByteBuffer buffer;
      private boolean closed = false;

      ByteBufferInputStream(ByteBuffer buffer) {
         this.buffer = buffer;
      }

      @Override
      public synchronized int read() throws IOException {
         if (closed) {
            throw new IOException("Stream already closed");
         }
         try {
            return buffer.get();
         } catch (BufferUnderflowException bue) {
            return -1;
         }
      }

      @Override
      public void close() throws IOException {
         super.close();
         closed = true;
      }
   }
}

You will want to override read(byte[], int, int) for efficiency. I also proposed this pull request to jclouds: https://github.com/apache/jclouds/pull/158 that you can improve on.

Upvotes: 0

Related Questions