CODEWITHSUNDEEP
node.jssocketsproxystreamrequest
iiAku
iiAku

Reputation: 11

NodeJS request pipe

I'm struggling with a technical issue, and because of I'm pretty new on NodeJS world I think I don't have the proper good practise and tools to help me solve this.

Using the well known request module, I'm making a stream proxy from a remote server to the client. Almost everything is fine and working properly until a certain point, if there is too much requests at the same time the server does no longer respond. Actualy it does get the client request but is unable to go through the stream process and serve the content.

What I'm currently doing:

  1. Creating a server with http module with http.createServer
  2. Getting remote url from a php script using exec
  3. Instanciate the stream

How I did it:

http://pastebin.com/a2ZX5nRr

I tried to investigate on the pooling stuff and did not understand everything, same thing the pool maxSocket was recently added, but did not helped me. I was also seting before the http.globalAgent to infinity, but I read that this was no longer limited in nodeJS from a while, so it does not help. See here: https://nodejs.org/api/http.html#http_http_globalagent

I also read this: Nodejs Max Socket Pooling Settings but I'm wondering what is the difference between a custom agent and the global one.

I believed that it could come from the server but I tested it on a very small one and a bigger one and it was not coming from there. I think it definitely coming from my app that has to be better designed. Indeed each time I'm restarting the app instance it works again. Also if I'm starting a fork of the server meanwhile the other is not serving anything on another port it will work. So it might not be about ressources.

Do you have any clue, tools or something that may help me to understand and debug what is going on?

NPM Module that can help handle stream properly: https://www.npmjs.com/package/pump

Upvotes: 0

Views: 2374

Answers (1)

iiAku
iiAku

Reputation: 11

I made few tests, and I think I've found what I was looking for. The unpipe things more info here: https://nodejs.org/api/stream.html#stream_readable_unpipe_destination

Can see and read this too, it leads me to understand few things about pipe remaining open when target failed or something: http://www.bennadel.com/blog/2679-how-error-events-affect-piped-streams-in-node-js.htm

So what I've done, i'm currently unpiping pipes when stream's end event is fired. However I guess you can make this in different ways, it depends on how you want to handle the thing but you may unpipe also on error from source/target.

Edit: I still have issues, it seams that the stream is now unpiping when it does not have too. I'll have to doubile check this.

Upvotes: 1

Related Questions