How to write to HDFS from a Spark Streaming script

Question

I am executing a Spark Streaming application and I want to dump some result to HDFS which is not in form of RDD ,they are simple strings . So how can I dump this data to HDFS , and if there is a way using which we can append these data to a file would be much helpful.

Answer 1

You always can create rdd from array of string: val newRDD = sc.parallelize(/* here your list or array */)

For writing output to file you can use saveAsTextFiles() method. foreachRDD + saveAsParquet(path, overwrite = true) - using this you can write each rdd to a single file.