CODEWITHSUNDEEP
javascriptnode.jsaxiosnode-streams
drmrbrewer
drmrbrewer

Reputation: 13009

Piping requests using gaxios (or axios)

At present I'm performing the trick of piping a request req to a destination url, and piping the response back to res, like so:

const request = require('request');

const url = 'http://some.url.com' + req.originalUrl;
const destination = request(url);

// pipe req to destination...
const readableA = req.pipe(destination);
readableA.on('end', function () {
    // do optional stuff on end
});

// pipe response to res...
const readableB = readableA.pipe(res);
readableB.on('end', function () {
    // do optional stuff on end
});

Since request is now officially deprecated (boo hoo), is this trick at all possible using the gaxios library? I thought that setting responseType: 'stream' on the request would do something similar as above, but it doesn't seem to work.

SImilarly, can gaxios be used in the following context:

request
.get('https://some.data.com')
.on('error', function(err) {
    console.log(err);
})
.pipe(unzipper.Parse())
.on('entry', myEntryHandlerFunction);

Upvotes: 14

Views: 18577

Answers (3)

Ringil
Ringil

Reputation: 6537

Looks like you are trying to basically forward requests from your express server to the clients. This worked for me.

import { request } from "gaxios";
const express = require("express");
const app = express();
const port = 3000;

app.get("/", async (req: any, res: any) => {
  const readableA = (await request({
    url: "https://google.com",
    responseType: "stream",
  })) as any;
  console.log(req, readableA.data);

  const readableB = await readableA.data.pipe(res);
  console.log(res, readableB);
});

app.listen(port, () => {
  console.log(`Example app listening at http://localhost:${port}`);
});

I imagine more complicated responses from A will require more nuiance in how to pipe it. But then you can probably just interact with express's response object directly and set the appropriate fields.

Upvotes: 1

Slava Dobromyslov
Slava Dobromyslov

Reputation: 3269

Install gaxios:

npm install gaxios

And then use request with the Readable type specified and with responseType set to 'stream'.

// script.ts

import { request } from 'gaxios';

(await(request<Readable>({
  url: 'https://some.data.com',
  responseType: 'stream'
}))
.data)
.on('error', function(err) {
    console.log(err);
})
.pipe(unzipper.Parse())
.on('entry', myEntryHandlerFunction);

// first-example.ts

import { request } from 'gaxios';

// pipe req to destination...
const readableA = (await request<Readable>({
  url: 'http://some.url.com',
  method: 'POST',
  data: req, // suppose `req` is a readable stream
  responseType: 'stream'
})).data;
readableA.on('end', function () {
    // do optional stuff on end
});

// pipe response to res...
const readableB = readableA.pipe(res);
readableB.on('end', function () {
    // do optional stuff on end
});

Gaxios is a stable tool and is used in official Google API client libraries. It's based on the stable node-fetch. And it goes with TypeScript definitions out of the box. I switched to it from the deprecated request and from the plain node-fetch library.

Upvotes: 2

Ashish Modi
Ashish Modi

Reputation: 7770

I guess if you provide responseType as stream and use res.data, you will get a stream which you could pipe like this

const {request} = require("gaxios");
const fs = require("fs");
const {createGzip} = require("zlib");
const gzip = createGzip();

(async () => {
  const res = await request({
    "url": "https://www.googleapis.com/discovery/v1/apis/",
    "responseType": "stream"
  });
  const fileStream = fs.createWriteStream("./input.json.gz");
  res.data.pipe(gzip).pipe(fileStream);
})();

Upvotes: 1

Related Questions