CODEWITHSUNDEEP
javagoogle-cloud-storagenio
oshai
oshai

Reputation: 15355

Stream content to Google Cloud Storage

I would like to upload a large Set<Integer> to Google Cloud Storage. I can do that with:

Blob result = storage.create(blobInfo, Joiner.on('\n').join(set).getBytes(UTF_8));

But this will create an intermediate String with all the content that might be too large.
I found an example with WriteChannel.write():

 Set<Integer> set = ...
 String bucketName = "my-unique-bucket";
 String blobName = "my-blob-name";
 BlobId blobId = BlobId.of(bucketName, blobName);
 byte[] content = Joiner.on('\n').join(set).getBytes(UTF_8);
 BlobInfo blobInfo = BlobInfo.newBuilder(blobId).setContentType("text/plain").build();
 try (WriteChannel writer = storage.writer(blobInfo)) {
     writer.write(ByteBuffer.wrap(content, 0, content.length));
 } catch (IOException ex) {
   // handle exception
 }

However, if I do that, the entire set is converted to a String and then to byte[]. The String itself might be too big.

Is there an example how to iterate over the set and transform it to a ByteBuffer? or should I do a loop on chunks of the set?

Upvotes: 0

Views: 1575

Answers (2)

Brandon Yarbrough
Brandon Yarbrough

Reputation: 38369

The most straightforward approach I could think of would be:

 try (WriteChannel writer = storage.writer(blobInfo)) {
   for(Integer val : set) {
     String valLine = val.toString() + '\n';
     writer.write(ByteBuffer.wrap(valLine.getBytes(UTF_8));
   }
 }

Mind you, this isn't very efficient. It creates a lot of small ByteBuffers. You could greatly improve on this by writing into a single larger ByteBuffer and periodically calling writer.write with it.

Upvotes: 1

Mike Schwartz
Mike Schwartz

Reputation: 12145

To avoid creating an intermediate String with all the bytes you can upload from a file. You can find example code to do an upload from a file in various languages here.

Upvotes: 0

Related Questions