CODEWITHSUNDEEP
javascalaapache-spark
pythonic
pythonic

Reputation: 21625

How to give file name when using NewHadoopRDD?

I am trying to use the newAPIHadoopRDD function in Spark, but I see that there is no argument in the constructor which takes file's name. So, how to set the filename in this case?

Upvotes: 0

Views: 152

Answers (1)

aa0eec7a
aa0eec7a

Reputation: 36

This is configured using org.apache.hadoop.conf.Configuration and details depend on the InputFormat you use. For example TextInputFormat uses mapred.input.dir.

In general if you want use file based source use newAPIHadoopFile.

Upvotes: 2

Related Questions