CODEWITHSUNDEEP
javaspark-streaming
kumar
kumar

Reputation: 71

How to use redis in Spark Streaming?

I am using spark streaming in java.I configured sparkconfig obj asSparkConf sparkConf = new SparkConf().setAppName("MyApp").setMaster("local[2]") .set("spark.streaming.stopGracefullyOnShutdown","true") .set("redis.host", "localhost") .set("redis.port", "6379"); and passing config obj in JavastreamingContext.

JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.milliseconds(1000));

how can i access redis with jssc object. thanks in advance.

Upvotes: 2

Views: 234

Answers (1)

fe2s
fe2s

Reputation: 435

This will create a stream from Redis list

  SparkConf sparkConf = new SparkConf().setAppName("MyApp").setMaster("local[2]")
            .set("spark.streaming.stopGracefullyOnShutdown", "true")
            .set("redis.host", "localhost")
            .set("redis.port", "6379");


    JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.milliseconds(1000));

    RedisConfig redisConfig = new RedisConfig(new RedisEndpoint(sparkConf));

    RedisStreamingContext redisStreamingContext = new RedisStreamingContext(jssc.ssc());
    String[] keys = new String[]{"myList"};
    RedisInputDStream<Tuple2<String, String>> redisStream =
            redisStreamingContext.createRedisStream(keys, StorageLevel.MEMORY_ONLY(), redisConfig);

    redisStream.print();

    jssc.start();
    jssc.awaitTermination();

Push some data to list:

LPUSH "myList" "aaaa"
LPUSH "myList" "bbbb"

Upvotes: 2

Related Questions