CODEWITHSUNDEEP
elixirphoenix-frameworkhttpoison
Junaid Farooq
Junaid Farooq

Reputation: 2598

%HTTPoison.Error{id: nil, reason: :closed}

We are using HTTPoison in our code base for the post and get requests while using the hackney pool. Posting here is just because I want to get some opinions from all the experts.

We are working with a lot of cameras, where, from each camera, we are getting images, And then all those images, we are saving them to seaweedFS servers.

We are making almost thousand of request on per second range. On the seaweedFS side, we have initiated the servers with a timeout of 10 seconds. And on our code base side, for uploading the image we are using this hackney configuration.

  :hackney_pool.child_spec(:seaweedfs_upload_pool, [timeout: 5000, max_connections: 1000]),
  :hackney_pool.child_spec(:seaweedfs_download_pool, [timeout: 5000, max_connections: 1000]),

For last few days, We are having an issue in this. At first, the timeout and max_connections on our side were 10 secs and 5K. And we started getting those HTTPoison errors as

{%HTTPoison.Error{id: nil, reason: :closed}}

The errors count was so high almost to 12K and so on, So for just for the time being, after discussing it in Elixir Slack channel I changed the timeout and max_connections to the above settings. After this, the errors reduced to 3K from 12K but only for 2 or 3 days, after that the error count is again going to 12K and so on.

The person who developed SeaweedFS, Pointed that the library we are using to send HTTP requests is not closing the connections, And on our seaweedFS server side(the server to which our code base send requests for reading and write) the result of this command is

root@Ubuntu-1404-trusty-64-minimal ~ # netstat | grep http | wc -l
5086

Can anyone suggest me something? What could be the reason of this error?

I have done few discussions on Elixir Slack Channel and what I come to know, with hackney pool, If we are making a post request with a timeout of 5 seconds and the HTTP request returned with success within 2 or 3 seconds then the connection which was made for this request will be remain open for other 2 seconds(completing 5 seconds) so that, that still open connection could be used by other http post request? Maybe I am wrong in this theory.

But It will be nice if anyone will point me to some better solutions with hackney pool.

We are using HTTPoison 0.11.1 which eventually using hackney 1.7.1.

Upvotes: 2

Views: 1393

Answers (1)

Pat Collins
Pat Collins

Reputation: 1

I encountered the following error:

 test login  --core test (Devato.Features.IndexPageTest)
 test/devato_web/features/index_page_test.exs:13
 ** (RuntimeError) Wallaby had an internal issue with HTTPoison:
 %HTTPoison.Error{id: nil, reason: :timeout}
 stacktrace:
   (wallaby 0.23.0) lib/wallaby/httpclient.ex:37: Wallaby.HTTPClient.make_request/5
   (wallaby 0.23.0) lib/wallaby/experimental/chrome.ex:96: Wallaby.Experimental.Chrome.start_session/1
   (wallaby 0.23.0) lib/wallaby.ex:78: Wallaby.start_session/1
   (devato 0.1.0) test/support/feature_case.ex:25: Devato.FeatureCase.__ex_unit_setup_0/1
   (devato 0.1.0) test/support/feature_case.ex:1: Devato.FeatureCase.__ex_unit__/2
   test/devato_web/features/index_page_test.exs:1: Devato.Features.IndexPageTest.__ex_unit__/2

I could run the tests in chrome headless, but i needed it to work so i can see it running on the browser. What worked for me was configuring the timeouts

Wallaby 0.23.0 added the following config:

config :wallaby,
    hackney_options: [timeout: :infinity, recv_timeout: :infinity]

Overriding a value

config :wallaby,
    hackney_options: [timeout: 5_000]

Upvotes: 0

Related Questions