CODEWITHSUNDEEP
javastreamjava-streaminputstream
BigBug
BigBug

Reputation: 6290

Stream<Object> to InputStream

How do i convert type Stream<Object> into an InputStream? Currently, I get the iterator and loop through all of the data converting it to a byteArray and adding it to an inputStream: 

 ByteArrayOutputStream bos = new ByteArrayOutputStream();
 ObjectOutputStream oos = new ObjectOutputStream(bos);

 Iterator<MyType> myItr = MyObject.getStream().iterator();

 while (myItr.hasNext()) {   

       oos.writeObject(myItr.next().toString()
         .getBytes(StandardCharsets.UTF_8));
   }
   oos.flush();
   oos.close();

   InputStream is = new ByteArrayInputStream(bao.toByteArray());

What is the overhead of doing this though? If my stream contains a terabyte of data, wouldn't I be sucking a terabyte of data into memory? Is there any better way to achieve this?

Upvotes: 4

Views: 7615

Answers (2)

Stephen James Hand
Stephen James Hand

Reputation: 172

Would this work for you?

https://gist.github.com/stephenhand/292cdd8bba7a452d83c51c00d9ef113c

It's an InputStream implementation that takes a Stream<byte[]> as input data. You just need to .map() your abitrary objects to byte arrays however you want each object to be represented as bytes.

It only calls a terminal operation on the Stream when the InputStream is read, pulling objects off the Stream as the consumer reads more of the InputStream so it never loads the whole set into memory

Upvotes: 2

shmosel
shmosel

Reputation: 50756

You should be able to convert the OutputStream into an InputStream using a pipe:

PipedOutputStream pos = new PipedOutputStream();
InputStream is = new PipedInputStream(pos);

new Thread(() -> {
    try (ObjectOutputStream oos = new ObjectOutputStream(pos)) {
        Iterator<MyType> myItr = MyObject.getStream().iterator();
        while (myItr.hasNext()) {
            oos.writeObject(myItr.next().toString()
                .getBytes(StandardCharsets.UTF_8));
        }
    } catch (IOException e) {
        // handle closed pipe etc.
    }
}).start();

Inspired by this answer.

Upvotes: 2

Related Questions