CODEWITHSUNDEEP
performanceapache-kafkaelixir
dauruy
dauruy

Reputation: 329

Write data come from concurrent requests to Kafka

I'm building an event collector, it will receive a http request like http://collector.me/?uuid=abc123&product=D3F4&metric=view then write request parameters to Apache Kafka topic, now I use Plug, Cowboy and KafkaEx.

defmodule Collector.Router do
  import Plug.Conn

  def init(opts) do
    opts
  end

  def call(conn, _opts) do
    conn = fetch_query_params(conn)
    KafkaEx.produce("test", 0, "#{inspect conn.query_params}")
    conn
    |> put_resp_content_type("text/plain")
    |> send_resp(200, "OK")
  end
end

AFAIK, Cowboy spawns a new process for each request, so I think write to Kafka in the call function is a proper way because it's easy to create hundreds of thousands of processes in Elixir. But I wonder if this is the right way to do? Do I need a queue before write to Kafka or something like that? My goal is handle as much concurrent requests as possible.

Thanks.

Upvotes: 0

Views: 349

Answers (1)

Hans Jespersen
Hans Jespersen

Reputation: 8335

Consider using the Confluent Kafka REST Proxy because then you might not need to write any server side code.

https://github.com/confluentinc/kafka-rest

Worst case is you might need to rewrite the incoming URL into a properly formatted HTTP POST with JSON data and the right HTTP header for Content-Type. This can be done with and application load balancer or a basic reverse Proxy like haproxy or nginx.

Upvotes: 1

Related Questions