CODEWITHSUNDEEP
apache-sparkapache-kafkaspark-streaming
HIREN GALA
HIREN GALA

Reputation: 109

How to set Kafka parameters from a properties file?

What I am doing currently is as below:

val topic = "mytopic"
val zkhosts = "localhost"
val zkports = "2181"

Setting it in my code and then sending it to kafkastream function works, but I want to read it from .properties file. Is there any possible solution?

Upvotes: 6

Views: 7660

Answers (1)

maasg
maasg

Reputation: 37435

Given this property file at /tmp/sample.properties

kafka.topic = "mytopic"
kafka.zkhost = "localhost"
kafka.zkports = 2191

We could use the plain-old java Property API to load the properties:

import java.io.FileReader
val configFile = new java.io.File("/tmp/sample.properties")
val reader = new FileReader(configFile)
val props = new Properties()
props.load(reader)
reader.close()

You could also use your favorite config library to load the property file as you would on any other program.

For example, you could use the popular typesafe config lib. There're many wrappers for Scala, but in its raw form you could do something like:

import com.typesafe.config.ConfigFactory
val configFile = new java.io.File("/tmp/sample.properties")
val kafkaConfig = ConfigFactory.parseFile(configFile)

import java.util.Properties
val kafkaProperties = new Properties()
kafkaProperties.put("zookeeper.hosts", kafkaConfig.getString("kafka.zkhost"))
kafkaProperties.put("zookeeper.port", kafkaConfig.getInt("kafka.zkports"):java.lang.Integer)
kafkaProperties.put("kafka.topic", kafkaConfig.getString("kafka.topic"))

(There're many ways to make that nice and compact. Here I'm using the most common form)

Upvotes: 5

Related Questions