CODEWITHSUNDEEP
javascripthtmlasynchronousasync-awaitfetch-api
Steven Lu
Steven Lu

Reputation: 43437

How to handle streaming data using fetch?

I have used async for with great success in handling output streams from processes with node.js, but I'm struggling to get something that I was hoping could "just work" with the browser fetch API.

This works great to async'ly handle chunks of output streaming from a process: 

for await (const out of proc.child.stdout) {
  ...
}

(in an async function context here of course)

I tried to do something similar in a browser where I want to gain access to the data while it is being sent to me from the server.

for await (const chunk of (await fetch('/data.jsonl')).body) {
  console.log('got', chunk);
}

This does not work in Chrome (Uncaught TypeError: (intermediate value).body is not async iterable).

For my use case this is not necessary, so I am simply using let data = await (await fetch(datapath)).text(); in my client code for now. This is analogous to the typical use of .json() instead of .text() on the awaited fetch, so no processing can begin until the entire response is received by the browser. This is not ideal for obvious reasons.

I was looking at Oboe.js (I think the relevant impl is somewhere near here) which pretty much deals with this but its internals are fairly ugly so it looks like that might be the only way to do this for now?

If async iteration isn't implemented (meaning async for cannot be used yet) isn't there another way to use the ReadableStream in a practical way?

Upvotes: 21

Views: 48263

Answers (3)

Bergi
Bergi

Reputation: 664589

According to the spec, a ReadableStream such as the fetch-API's Response.body does have a values method. For some reason it's not async-iterable itself, you explicitly have to call that method:

const response = await fetch('/data.json');
if (!response.ok)
    throw new Error(await response.text());
for await (const chunk of response.body.values()) {
    console.log('got', chunk);
}

As of August 2024, this is implemented in Chrome, Edge and Firefox, but not Safari (see MDN: response.body as ReadableStream, ReadableStream.values).

Upvotes: 8

Domenic
Domenic

Reputation: 112837

Unfortunately async iterable support is not yet implemented, despite being in the spec. Instead you can manually iterate, as shown in this example from the spec. (I'll convert examples to async/await for you in this answer.)

const reader = response.body.getReader();
const { value, done } = await reader.read();

if (done) {
  console.log("The stream was already closed!");
} else {
  console.log(value);
}

You can use recursion or a loop to do this repeatedly, as in this other example:

async function readAllChunks(readableStream) {
  const reader = readableStream.getReader();
  const chunks = [];
  
  let done, value;
  while (!done) {
    ({ value, done } = await reader.read());
    if (done) {
      return chunks;
    }
    chunks.push(value);
  }
}

console.log(await readAllChunks(response.body));

Upvotes: 35

Steven Lu
Steven Lu

Reputation: 43437

I believe the current state of affairs in mid 2020 is that async for does not work on the fetch body yet.

https://github.com/whatwg/streams/issues/778 This issue appears to have tracking bugs for browsers and none of them have the functionality implemented yet.

I don't currently know of another way to make use of the .body ReadableStream provided by fetch.

The standard way to do the task implicit in the question is to use a websocket.

Upvotes: 3

Related Questions