CODEWITHSUNDEEP
apache-sparkhadooppysparkhdfshdp
neorus
neorus

Reputation: 494

Incomplete HDFS URI, no host, altohugh file does exist

i'm trying to access via pyspark to my files in hdfs with the following code:

spark = SparkSession.builder.appName("MongoDBIntegration").getOrCreate()
receipt = spark.read.json("hdfs:///bigdata/2.json")

and i get an error Incomplete HDFS URI, no host: hdfs:///bigdata/2.json

but if i write the command hdfs dfs -cat /bigdata/1.json it does print me my file

Upvotes: 0

Views: 3425

Answers (1)

vinsce
vinsce

Reputation: 1338

The error message says that you have not specified the host in the HDFS URI.

Try to change the URI with:

hdfs://<HDFS_HOST>:<HDFS_PORT>/bigdata/2.json

Upvotes: 2

Related Questions