CODEWITHSUNDEEP
apache-sparkapache-spark-sqlspark-streamingmqttmosquitto
gambiteer
gambiteer

Reputation: 33

Spark 3.0 - Read data from an MQTT stream

I would like to read data stream from a mosquitto based MQTT topic into my Spark 3.0 application. I tried with the Bahir library the following way:

Dataset<Row> df = sparkSession
           .readStream()
           .format("org.apache.bahir.sql.streaming.mqtt.MQTTStreamSourceProvider")
           .option("topic", "mytopic")
           .load("tcp://localhost:1883");

Unfortunately this has failed:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/StreamWriteSupport

Apparently this only works with the 2.x Spark. Any idea how to make this working with Spark 3 (or with a different lib)?

Upvotes: 0

Views: 684

Answers (1)

Michael Heil
Michael Heil

Reputation: 18515

Looking at the available libraries here, (see picture below) this library is not available for Spark 3.x. Also, I am not aware of any other public library.

enter image description here

Upvotes: 1

Related Questions