CODEWITHSUNDEEP
twitterapache-kafkaapache-flinkflink-streaming
Catarina Nogueira
Catarina Nogueira

Reputation: 1138

Picks on latency graph of a Flink/Kafka application

I have a application that receives tweets from a Kafka topic, have a window of one second, and then save these tweets on Cassandra by a AsyncIO operation that allows to open maximum of 100 threads(last parameter of AsyncIO operator) without doing any pre-processing on the data: just save tweet by tweet with a timestamp of when it was saved.

Then, I have stressed the Flink application sending 3 million tweets and did a graph in Grafana that shows how many tweets are being saved on the database, but this graph show some picks, is not a continuous line, and I can't understand why.

So you can see that in an interval of one minute it saves 7k then goes to 5k and then to 2k for example. How can I find out why?

enter image description here

Upvotes: 0

Views: 122

Answers (1)

Arvid Heise
Arvid Heise

Reputation: 3634

First of all, if you want to write to cassandra, I'd use the connector. Implementing something like exactly-once manually correctly is very hard if not close to impossible.

Second, AsyncIO is not starting 100 threads. In fact, it's not starting any threads for users. You need to start them yourself through any means. Usually, it's using the callback mechanism of the external systems where the libraries have their own connection pool.

If you are making synchronize calls, you need to manage your own thread pool. I recommend using Executors.newCachedThreadPool() and submitting your async tasks to it. AsyncIO will only help to merge the async results back into a synchronous stream.

Third, 100 threads might be quite a lot, depending on your setup. Also note that if you use Flink's scale-up (using more than one slot per taskmanager) would multiply the used threads.

Upvotes: 1

Related Questions