CODEWITHSUNDEEP
apache-sparkstreamingvisualization
Anshuman Panwar
Anshuman Panwar

Reputation: 137

spark streaming visualization

I am using spark streaming to stream data from kafka broker. I am performing transformations on the data using spark streaming. Can someone suggest a visualization tool which I can use to show real-time graphs and charts which update as data streams in?

Upvotes: 2

Views: 3766

Answers (5)

rshah
rshah

Reputation: 1

Check out Lightning: A Data Visualization Server
https://github.com/lightning-viz/lightning
The server is designed to for making web-based interactive visualizations using D3. It is designed for large data sets and continuously updating data streams.

Upvotes: 0

pkwied
pkwied

Reputation: 31

You can use Pro BI Tools like Tableau, Power BI or even MS Excel.. For testing, I use MS Excel with 1 min auto refresh.

You can also write python code for this.

Upvotes: 0

desaiankitb
desaiankitb

Reputation: 1052

Apart from looking at spark's own streaming UI tab, I highly recommend use of graphite sinks. Spark streaming is a long running application so for monitoring purposes this can be really handy.

In no time using graphite dashboards you will kick start monitoring your spark streaming application.

The best literature I know is, here in section monitoring. and [here too] (https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/)

It provides configuration and other details. Some of the dashboards you will find ready-made in json format on some or other github links but again I found these two posts most useful in my production application.

I hope this will help you for visualizing and monitoring your application internals in spark streaming application.

Upvotes: 1

slimoo
slimoo

Reputation: 123

You could store your results in ElasticSearch and then use Kibana to perform visualizations.

Upvotes: 1

Sumit
Sumit

Reputation: 1420

you have use Websockets for building real-time streaming Graphs. As such there are no BI tools but there are JS libraries which can help in building real-time graphs - http://www.pubnub.com/blog/tag/d3-js/

Upvotes: 0

Related Questions