CODEWITHSUNDEEP
javascriptnode.jshttphttpsbuffer
Jacob
Jacob

Reputation: 1835

Response Buffer data is truncated in Node JS

I'm trying to figure out how to use the request function of the http and https built-in modules in Node JS (I am aware of third-party wrappers but I'm trying to figure it out myself). I'm running into an issue where the Buffer data from the Response is partially cut off at the end. The issue does not occur when testing with cURL.

Here is the code I've been using:

const { request: httpRequest } = require("http");
const { request: httpsRequest } = require("https");
const parseURLData = (url, init) => {
  const { hostname, protocol, port: somePort, pathname, search } = new URL(url);
  const port = +somePort || (protocol === "https:" ? 443 : 80);
  const options = { hostname, port, path: pathname + search, ...init };
  return [protocol, options];
};
const makeRequest = (url, init = { method: "GET" }) => {
  const [protocol, options] = argumentsToOptions(url, init);
  const request = protocol === "http:" ? httpRequest : httpsRequest;
  return new Promise((resolve, reject) =>
    request(options, (res) => {
      res.on("error", reject);
      resolve(res);
    })
      .on("error", reject)
      .end()
  );
};
// not using `async/await` while testing
makeRequest("https://jsonplaceholder.typicode.com/users/1/")
  .then((res) => 
    new Promise((resolve) =>
      res.on("data", (buffer) => {
        resolve(buffer.toString("utf8")); // part of data is cut off
        // resolve(JSON.parse(buffer.toString()));
      })
    )
  )
  .then(console.log)
  .catch(console.error);

Here is the expected output (from cURL):

{
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret",
  "email": "[email protected]",
  "address": {
    "street": "Kulas Light",
    "suite": "Apt. 556",
    "city": "Gwenborough",
    "zipcode": "92998-3874",
    "geo": {
      "lat": "-37.3159",
      "lng": "81.1496"
    }
  },
  "phone": "1-770-736-8031 x56442",
  "website": "hildegard.org",
  "company": {
    "name": "Romaguera-Crona",
    "catchPhrase": "Multi-layered client-server neural-net",
    "bs": "harness real-time e-markets"
  }
}

And here is the actual output, which for some reason is slightly different each time I run the code:

{
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret",
  "email": "[email protected]",
  "address": {
    "street": "Kulas Light",
    "suite": "Apt. 556",
    "city": "Gwenborough",
    "zipcode": "92998-3874",
    "geo": {
      "lat": "-37.3159",
      "lng": "81.1496"
    }
  },
  "p

What is the correct solution to this problem? How can I get the all of the data from the request?

Upvotes: 3

Views: 2362

Answers (1)

Gamedroit
Gamedroit

Reputation: 385

The buffer should only be processed when the end event fires, otherwise you may be processing an incomplete buffer.

makeRequest("https://jsonplaceholder.typicode.com/users/1/")
  .then((res) => 
    new Promise((resolve) => {
        let totalBuffer = "";

        res.on("data", (buffer) => {
            totalBuffer += buffer.toString("utf8");    
        });

        res.on("end", () => resolve(totalBuffer));
    })
  )
  .then(console.log)
  .catch(console.error);

The response is almost always truncated into several pieces when the file exceeds 1mb, so it is necessary to use the end event which indicates that all available data has been handled by the stream.

https://nodejs.org/api/http.html Look for "JSON fetching example"

Upvotes: 5

Related Questions