CODEWITHSUNDEEP
node.jstypescriptmultipartform-data
cyberwombat
cyberwombat

Reputation: 40124

Unable to use createReadStream with Node 18 FormData

Previously to Node 18 releasing fetch/FormData we could do:

import FormData from 'form-data'

const form = new FormData();
form.append('my_field', 'my value');
form.append('my_buffer', new Buffer(10));
form.append('my_file', fs.createReadStream('/foo/bar.jpg'));

However with the global FormData I can no longer pass a stream - the error at the "my_file" line is:

Argument of type 'ReadStream' is not assignable to parameter of type 'string | Blob'

I know this is still experimental so potentially a bug or is there an alternative way to do this - besides reading the entire file as a string...

Upvotes: 12

Views: 9651

Answers (5)

bobbyg603
bobbyg603

Reputation: 3850

Node.js 19.8+

You can use fs.openAsBlob.

import { openAsBlob } from 'node:fs/promises';
const blob = await openAsBlob('the.file.txt');

Then, you can append the file-backed blob to your form data.

const formData = new FormData();
formData.append('file', blob, name);

Node.js < 19.8

We were unable to find a CommonJS-compatible streaming Blob implementation, so we took inspiration from @roman-rodionov's answer and created a library that will create a streamable File. If using Node.js@<18.15 you'll also need to polyfill File.

You can install node-streamable-file to do this.

npm i node-streamable-file

We made a few tweaks to the implementation, most notably we removed the extra class definition. Here's our implementation if you'd like to do streaming file uploads without adding a dependency.

import { File } from 'buffer';
import { open } from 'node:fs/promises';
import { basename } from 'node:path';

export async function createStreamableFile(path: string): Promise<File> {
    const name = basename(path);
    const handle = await open(path);
    const { size } = await handle.stat();

    const file = new File([], name);
    file.stream = () => handle.readableWebStream();

    // Set correct size otherwise, fetch will encounter UND_ERR_REQ_CONTENT_LENGTH_MISMATCH
    Object.defineProperty(file, 'size', { get: () => size });

    return file;
}

Use createStreamableFile like so

const formData = new FormData();
const file = await createStreamableFile('path/to/file');
formData.append('file', file);

await fetch(url, {
  method: 'POST',
  body: formData
});

Additionally, if you're using TypeScript there's an issue with the File and Blob types. You can fix the problem by casting to the instance of File to unknown and then to Blob.

Upvotes: 2

Endless
Endless

Reputation: 37885

The correct way

The most straight forward way without any dependencies is to use fs.openAsBlob(path, {type})
It will return a Promise<Blob> and this also require NodeJS v20+

You don't actually need a File instance cuz FormData.append() accepts blobs too. fd.append('field', blob, filename). fyi, This also works in browsers.

import fs from 'node:fs'

const pkg = await fs.openAsBlob('./package.json')
const blob = await fs.openAsBlob('./README.md', { type: 'text/plain' })

// You don't really need a File (see example below by appending blobs)
const file = new File([pkg], 'package.json', { type: 'application/json' })

const fd = new FormData()
fd.append('files', blob, 'README.md') // Blobs works as well
fd.append('files', file)

I would say this is the proper way to get hold of a blob without reading any of the content into memory. or by extending files and overriding the stream method with a custom function, cuz that's wrong. and should best be avoided.

The node-fetch way.

If you are using node-fetch then it already comes with ways to create blob from paths. and it works all the way from NodeJS v14+

import fetch, {
  Blob, blobFrom, blobFromSync,
  File, fileFrom, fileFromSync,
  FormData
} from 'node-fetch'

const fd = new FormData()
fd.append('files', fileFromSync('./README.md'))
fd.append('files', await fileFrom('./package.json', 'text/plain'))

This utility method dose not really come from node-fetch, but actually from fetch-blob. so this blobFrom and fileFrom can be used without installing node-fetch

import {
  Blob, blobFrom, blobFromSync,
  File, fileFrom, fileFromSync, 
  createTemporaryBlob, createTemporaryFile 
} from 'fetch-blob/from.js'

Upvotes: -1

Max
Max

Reputation: 1

@eliw00d regarding the native node fetch implementation suggested by @romanr: I stumbled across the same issue (Please close FileHandle objects explicitly), and was also not able to create a new ReadableStream that calls the handle.close() method as suggested. My solution was to import {finished} from 'node:stream' and then add it:

file.stream = function () {
    const webStream = handle.readableWebStream();
    // Ensure that the handle gets closed!
    const cleanup = finished(webStream, (_err) => {
        handle.close();
        cleanup();
    });
    return webStream;
};

This seems to work.

Upvotes: -1

Roman Rodionov
Roman Rodionov

Reputation: 69

Here is the implementation of pure streaming without reading the entire content in memory.

Node.JS built-in API:


import { open } from 'node:fs/promises';
import { File } from 'buffer';
  
const handle = await open('/path/to/your/file');

const stat = await handle.stat();

class MyFile extends File {
    // we should set correct size
    // otherwise we will encounter UND_ERR_REQ_CONTENT_LENGTH_MISMATCH
    size = stat.size;
}

const file = new MyFile([], 'file-name')


file.stream = function() {
    return handle.readableWebStream();
};

const formData = new FormData();

formData.append('file_key', file);

fetch('http://localhost', {
    method: 'post',
    body: formData
});

Using node-fetch:

import * as fs from 'fs';
import fetch, { FormData, File  } from 'node-fetch';

const stream = fs.createReadStream('/path/to/your/file');
const stat = fs.statSync('/path/to/your/file');

class MyFile extends File {
    size = stat.size;
}

const file = new MyFile([], 'file-name');

file.stream = function() {
    return stream;
};

const formData = new FormData();

formData.append('file_key', file);

fetch('http://localhost', {
    method: 'post',
    body: formData
});

Upvotes: 5

Phil
Phil

Reputation: 164895

Node v18's native FormData is an implementation of the w3 FormData interface so you need to use that API.

The append() method accepts a Blob so you should be able to use the blob stream consumer

import { createReadStream } from 'node:fs';
import { blob } from 'node:stream/consumers';

// assuming a valid async context for brevity
const file = await blob(createReadStream("/foo/bar.jpg"));

const formData = new FormData();
formData.append("my_file", file, "bar.jpg");

Upvotes: 1

Related Questions