CODEWITHSUNDEEP
httpapache-stormning
user983022
user983022

Reputation: 979

How to cluster multiple HTTP Clients?

I have a use case which involves pulling/streaming data from numerous HTTP endpoints, in excess of over 100.

I have a standalone java app that can manage up to 30+ client requests using ning async client http library but was looking for some ideas on how I could scale this up to handle much more.

The use case is to pull the data from the end points and push them into a kafka queue (similar to jms queue) for processing by a storm topology. The bit I'm stuck on is how to best efficiently get the http end point data into the Kafka queues in the first place.

thanks

Upvotes: 1

Views: 123

Answers (0)

Related Questions