CODEWITHSUNDEEP
javascriptangularangular-httpclient
maersk
maersk

Reputation: 103

Stream large blob file using StreamSaver.js

I'm trying to download a large data file from a server directly to the file system using StreamSaver.js in an Angular component. But after ~2GB an error occurs. It seems that the data is streamed into a blob in the browser memory first. And there is probably that 2GB limitation. My code is basically taken from the StreamSaver example. Any idea what I'm doing wrong and why the file is not directly saved on the filesystem?

Service:

public transferData(url: string): Observable<Blob> {
    return this.http.get(url, { responseType: 'blob' });
}

Component:

download(url: string) {
    this.extractionService.transferData(url)
      .subscribe(blob => {
        const fileStream = streamSaver.createWriteStream('data.tel', {
          size: blob.size
        });
        const readableStream = blob.stream();
        if (window.WritableStream && readableStream.pipeTo) {
          return readableStream
            .pipeTo(fileStream)
            .then(() => console.log("done writing"));
        }
        const writer = fileStream.getWriter();
        const reader = readableStream.getReader();
        const pump = () =>
          reader.read()
            .then(res => res.done ? writer.close() : writer.write(res.value).then(pump));
        pump();
      });
}

The header of the requested file:
"Content-Type: application/octet-stream\r\n"
"Content-Disposition: attachment; filename=data.tel\r\n"

Upvotes: 3

Views: 11233

Answers (3)

Александр Калугин
Александр Калугин

Reputation: 36

this.http.post(`url`, body, { responseType: 'blob'}).pipe(
  concatMap((response) => {
    const fileStream = streamSaver.createWriteStream('someFile');
    return response.body.stream().pipeTo(fileStream);
  })
);

Upvotes: 1

Triet Doan
Triet Doan

Reputation: 12085

I was more or less in the same situation before. Angular http service won't work in this use case since it does not give you a ReadableStream. My solution is using fetch API instead.

But be careful that Streaming response body of fetch is an experimental feature and not compatible with all browsers. According to my test, it works fine on Google Chrome but not with Firefox or Safari. To overcome this limitation, I use a Javascript library called web-streams-polyfill together with fetch.

The code looks somehow like this:

import { WritableStream } from 'web-streams-polyfill/ponyfill';
import streamSaver from 'streamsaver';

fetch(url, {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify(data)
})
.then(response => {

    let contentDisposition = response.headers.get('Content-Disposition');
    let fileName = contentDisposition.substring(contentDisposition.lastIndexOf('=') + 1);

    // These code section is adapted from an example of the StreamSaver.js
    // https://jimmywarting.github.io/StreamSaver.js/examples/fetch.html

    // If the WritableStream is not available (Firefox, Safari), take it from the ponyfill
    if (!window.WritableStream) {
        streamSaver.WritableStream = WritableStream;
        window.WritableStream = WritableStream;
    }

    const fileStream = streamSaver.createWriteStream(fileName);
    const readableStream = response.body;

    // More optimized
    if (readableStream.pipeTo) {
        return readableStream.pipeTo(fileStream);
    }

    window.writer = fileStream.getWriter();

    const reader = response.body.getReader();
    const pump = () => reader.read()
        .then(res => res.done
            ? writer.close()
            : writer.write(res.value).then(pump));

    pump();
})
.catch(error => {
    console.log(error);
});;

The idea is to check if window.WritableStream is available in the current browser or not. If not, assign the WritableStream from ponyfill directly to streamSaver.WritableStream property.

Since I faced this problem some time ago, my solution was only tested on Google Chrome 78, Firefox 70, Safari 13; web-streams-polyfill 2.0.5, and StreamSaver.js 2.0.3

Upvotes: 5

Endless
Endless

Reputation: 37885

Suggestion / Background

StreamSaver is targeted for those who generate large amount of data on the client side, like a long camera recording for instance. If the file is coming from the cloud and you already have a Content-Disposition attachment header then the only thing you have to do is to open this URL in the browser.

There is a few ways to download the file:

  • location.href = url
  • <a href="url">download</a>
  • <iframe src="url" hidden>
  • and for those who need to post data or use a other HTTP method, they can post a (hidden) <form> instead.

As long as the browser does not know how to handle the file then it will trigger a download instead, and that is what you are already doing with Content-Type: application/octet-stream

Since you are downloading the file using Ajax and the browser knows how to handle the data (giving it to main JS thread), then Content-Type and Content-Disposition don't serve any purpose.

StreamSaver tries to mimic how the server saves files with ServiceWorkers and custom responses.
You are already doing it on the server! The only thing you have to do is stop using AJAX to download files. So I don't think you will need StreamSaver at all.

Your problem

... is that you first download the whole data into memory as a Blob first and then you save the file. This defeats the whole purpose of using StreamSaver, then you could just as well use the simpler FileSaver.js library or manually create an object url + link from a Blob like FileSaver.js does.

Object.assign(
  document.createElement('a'), 
  { href: URL.createObjectURL(blob), download: 'name.txt' }
).click()

Besides, you can't use Angular's HTTP service, since they use the old XMLHttpRequest instead, and it can't give you a ReadableStream like fetch does from response.body so my advice is to just simply use the Fetch API instead.

https://github.com/angular/angular/issues/36246

Upvotes: 7

Related Questions