Retrieve data from a ReadableStream object?

Question

How may I get information from a ReadableStream object?

I am using the Fetch API and I don't see this to be clear from the documentation.

The body is being returned as a ReadableStream and I would simply like to access a property within this stream. Under Response in the browser dev tools, I appear to have this information organised into properties, in the form of a JavaScript object.

fetch('http://192.168.5.6:2000/api/car', obj)
    .then((res) => {
        if(!res.ok) {
            console.log("Failure:" + res.statusText);
            throw new Error('HTTP ' + res.status);
        } else {
            console.log("Success :" + res.statusText);
            return res.body // what gives?
        }
    })

Answer 1

You may have asked the wrong question to solve your problem, but here is an answer to your actual question. An inspiration may be the source code of the Node.js stream/consumers module.

res.body is a ReadableStream that emits chunks as Uint8Arrays. Note that ReadableStream objects created elsewhere may emit other data types than Uint8Array, and the methods outlined in this answer need to be adjusted in those cases.

There are multiple ways to consume such a stream:

Convert a stream to a Uint8Array

Using new Response(stream).arrayBuffer()

If you want to retrieve the whole content of the stream in one go, the easiest way would is to wrap it in a Response object. You can then use one of the several methods to retrieve the object as a string, a JSON object, an array buffer or something else. For example, to retrieve it as an array buffer:

export async function streamToArrayBuffer(stream: ReadableStream<Uint8Array>): Promise<Uint8Array> {
   return new Uint8Array(await new Response(stream).arrayBuffer());
}

Note that new Response(stream) only works for Uint8Array streams. If the stream emits any other type (such as strings), it will result in a TypeError: Received non-Uint8Array chunk error.

Also note that if an error occurs in the stream, this method will throw a TypeError: Failed to fetch error without a stack trace! If you want proper error handling, use one of the other methods.

Using stream.getReader()

The following function will collect all the chunks in a single Uint8Array:

function concatArrayBuffers(chunks: Uint8Array[]): Uint8Array {
    const result = new Uint8Array(chunks.reduce((a, c) => a + c.length, 0));
    let offset = 0;
    for (const chunk of chunks) {
        result.set(chunk, offset);
        offset += chunk.length;
    }
    return result;
}

export async function streamToArrayBuffer(stream: ReadableStream<Uint8Array>): Promise<Uint8Array> {
    const chunks: Uint8Array[] = [];
    const reader = stream.getReader();
    while (true) {
        const { done, value } = await reader.read();
        if (done) {
            break;
        } else {
            chunks.push(value);
        }
    }
    return concatArrayBuffers(chunks);
}

Using async iterator

ReadableStream implements the async iterator protocol. However, this is not supported by most browsers yet, but in Node.js you can already use it (using TypeScript, you will have to use the NodeJS.ReadableStream interface, see this discussion).

The following code will collect all the chunks into a single Uint8Array:

export async function streamToArrayBuffer(stream: ReadableStream<Uint8Array>): Promise<Uint8Array> {
    const chunks: Uint8Array[] = [];
    for await (const chunk of stream) {
        chunks.push(chunk);
    }
    return concatArrayBuffers(chunks);
}

In the future when browser support Array.fromAsync(), this can be shortened to:

export async function streamToArrayBuffer(stream: ReadableStream<Uint8Array>): Promise<Uint8Array> {
    return concatArrayBuffers(await Array.fromAsync(stream));
}

Convert a stream to a string

Using new Response(stream).text()

Just like described above for an array buffer, a stream of Uint8Array can be converted to a string by using Response.text():

export async function streamToString(stream: ReadableStream<Uint8Array>): Promise<string> {
   return await new Response(stream).text();
}

Here again, note that this will fail with a generic exception when an error occurs on the stream, making debugging difficult.

Using TextDecoderStream

TextDecoderStream will convert the stream of Uint8Array chunks into a stream of string chunks. This way you can collect the contents of a stream as a string directly. Note that browser support in Firefox has only been added in September 2022, so you might not want to use this in production just yet.

export async function streamToText(stream: ReadableStream<Uint8Array>): Promise<string> {
    let result = '';
    const reader = stream.pipeThrough(new TextDecoderStream()).getReader();
    while (true) {
        const { done, value } = await reader.read();
        if (done) {
            break;
        }

        result += value;
    }
    return result;
}

In browsers that support it, you can also consume this stream of strings using the async iterator protocol:

export async function streamToText(stream: ReadableStream<Uint8Array>): Promise<string> {
    let result = '';
    for (const chunk of stream.pipeThrough(new TextDecoderStream()).getReader())
        result += chunk;
    }
    return result;
}

Or in browsers that support Array.fromAsync(), even shorter:

export async function streamToText(stream: ReadableStream<Uint8Array>): Promise<string> {
    const chunks = await Array.fromAsync(stream.pipeThrough(new TextDecoderStream()).getReader()));
    return chunks.join("");
}

Using TextDecoder

To convert the Uint8Arrays generated by some of the functions above to a string, you can then use TextDecoder:

const buffer = await streamToArrayBuffer(res.body);
const text = new TextDecoder().decode(buffer);

Note that this should only been used on the whole content, not on individual Uint8Array chunks, as some characters may consist of multiple bytes and might be split up between chunks.

Convert a stream to a JSON object

Using new Response(stream).json()

Just like described above for an array buffer and a string, a stream of Uint8Array can be parsed as JSON by using Response.json():

export async function streamToJson(stream: ReadableStream<Uint8Array>): Promise<unknown> {
   return await new Response(stream).json();
}

Using JSON.parse()

Use any of the methods above to convert the stream to a string and then use JSON.parse() to parse that string.

Answer 2

For those who have a ReadableStream and want to get the text out of it, a short hack is to wrap it in a new Response (or Request) and then use the text method:

let text = await new Response(yourReadableStream).text();
// or:
let json = await new Response(yourReadableStream).json();

Answer 3

If you are using React Native, it used to not be possible to do this.

But streaming is now possible with https://github.com/react-native-community/fetch.

This was actually a bug that was never addressed by RN team for a while, and this repo emerged to provide a better fetch that complies with WHATWG Spec

This is a fork of GitHub's fetch polyfill, the fetch implementation React Native currently provides. This project features an alternative fetch implementation directy built on top of React Native's Networking API instead of XMLHttpRequest for performance gains. At the same time, it aims to fill in some gaps of the WHATWG specification for fetch, namely the support for text streaming.

Here's how to use it:

Install

This concise steps are from hours of debugging, and I dont want to waste your time.

$ npm install react-native-fetch-api --save

Now install polyfills:

$ npm install react-native-polyfill-globals

Use the polyfill with fetch:

Add the following code to the top of your app's entry file, index.js, located at the root of your project. Now your new Fetch is available globally.

import { polyfill as polyfillFetch } from 'react-native-polyfill-globals/src/fetch';
polyfill();

Now you can use the stream object like the normal browser fetch. Make sure to specify the option textStreaming true.

fetch('https://jsonplaceholder.typicode.com/todos/1', { reactNative: { textStreaming: true } })
  .then(response => response.body)
  .then(stream => ...)

Hope this helps!

Answer 4

const resText = await response.json(); //returns string

const resJson = JSON.parse(resText); //objectified

sometimes (especially when you are using an api written in another language), response.json returns string. You should treat it like res.body and parse it, then you have the object. Hope it helps.

Answer 5

here is how I implemented it. In this case the api is returning a ndjson as a stream, and I am reading it in chunks. In ndjson format, data is split by new lines, so each line by itself is a basic json which I parsed and added to fetchedData variable.

var fetchedData = [];

fetch('LinkGoesHere', {
    method: 'get',
    headers: {
        'Authorization': 'Bearer TokenGoesHere' // this part is irrelevant and you may not need it for your application
    }
})
.then(response => {
    if (!response.ok) {
        throw new Error(`HTTP error! Status: ${response.status}`);
    }
    return response.body.getReader();
})
.then(reader => {
    let partialData = '';

    // Read and process the NDJSON response
    return reader.read().then(function processResult(result) {
        if (result.done) {
            return;
        }

        partialData += new TextDecoder().decode(result.value, { stream: true });
        const lines = partialData.split('\n');

        for (let i = 0; i < lines.length - 1; i++) {
            const json = JSON.parse(lines[i]);
            fetchedData.push(json); // Store the parsed JSON object in the array
        }

        partialData = lines[lines.length - 1];

        return reader.read().then(processResult);
    });
})
.then(() => {
    // At this point, fetchedData contains all the parsed JSON objects
    console.log(fetchedData);
})
.catch(error => {
    console.error('Fetch error:', error);
});

Answer 6

In order to access the data from a ReadableStream you need to call one of the conversion methods (docs available here).

As an example:

fetch('https://jsonplaceholder.typicode.com/posts/1')
  .then(function(response) {
    // The response is a Response instance.
    // You parse the data into a useable format using `.json()`
    return response.json();
  }).then(function(data) {
    // `data` is the parsed version of the JSON returned from the above endpoint.
    console.log(data);  // { "userId": 1, "id": 1, "title": "...", "body": "..." }
  });

---

EDIT: If your data return type is not JSON or you don't want JSON then use text()

As an example:

fetch('https://jsonplaceholder.typicode.com/posts/1')
  .then(function(response) {
    return response.text();
  }).then(function(data) {
    console.log(data); // this will be a string
  });

Answer 7

response.json() returns a Promise. Try ...

res.json().then(body => console.log(body));

where response is the result of the fetch(...)

Answer 8

I just had the same problem for over 12 hours before reading next, just in case this helps anyone. When using nextjs inside your _api page you will need to use JSON.stringify(whole-response) and then send it back to your page using res.send(JSON.stringify(whole-response)) and when it's received on the client side you need to translate it back into json format so that it's usable. This can be kinda figured out by reading their serialization section. Hope it helps.

Answer 9

Note that you can only read a stream once, so in some cases, you may need to clone the response in order to repeatedly read it:

fetch('example.json')
  .then(res=>res.clone().json())
  .then( json => console.log(json))

fetch('url_that_returns_text')
  .then(res=>res.clone().text())
  .then( text => console.log(text))

Answer 10

Little bit late to the party but had some problems with getting something useful out from a ReadableStream produced from a Odata $batch request using the Sharepoint Framework.

Had similar issues as OP, but the solution in my case was to use a different conversion method than .json(). In my case .text() worked like a charm. Some fiddling was however necessary to get some useful JSON from the textfile.

Answer 11

I dislike the chaining thens. The second then does not have access to status. As stated before 'response.json()' returns a promise. Returning the then result of 'response.json()' in a acts similar to a second then. It has the added bonus of being in scope of the response.

return fetch(url, params).then(response => {
    return response.json().then(body => {
        if (response.status === 200) {
            return body
        } else {
            throw body
        }
    })
})

Answer 12

If you just want the response as text and don't want to convert it into JSON, use https://developer.mozilla.org/en-US/docs/Web/API/Body/text and then then it to get the actual result of the promise:

fetch('city-market.md')
  .then(function(response) {
    response.text().then((s) => console.log(s));
  });

or

fetch('city-market.md')
  .then(function(response) {
    return response.text();
  })
  .then(function(myText) {
    console.log(myText);
  });

Answer 13

Some people may find an async example useful:

var response = await fetch("https://httpbin.org/ip");
var body = await response.json(); // .json() is asynchronous and therefore must be awaited

json() converts the response's body from a ReadableStream to a json object.

The await statements must be wrapped in an async function, however you can run await statements directly in the console of Chrome (as of version 62).