CODEWITHSUNDEEP
jsondataframepysparkdatabricks
Reut
Reut

Reputation: 1592

IllegalArgumentException: Path must be absolute when trying to read json in databricks

I have jsut started to use databricks, I'm using the community cloud and I'm trying to read json file. I have tried to do this as following:

from pyspark.sql import SparkSession

df=spark.read.json('people')

But then I'm getting error:

IllegalArgumentException: Path must be absolute: people

I have tried to acess the people.json file with different writing:

'people.json', '/people.json','Data/deafult/people' ect., the last one is based on where I understood that the data is stored:

enter image description here

However, that did not work and I kept geting the same error message.

HoWhenever I try to read it with sql.Context.sql it shows the table:

df=sqlContext.sql("SELECT * FROM people")
df.show()

>>>
+----+-------+
| age|   name|
+----+-------+
|null|Michael|
|  30|   Andy|
|  19| Justin|
+----+-------+

So my question is how can I open the json file with spark.read.json?

Upvotes: 0

Views: 19302

Answers (3)

Lex D
Lex D

Reputation: 11

You might need to try:

df=spark.read.json('default.people')

But as noted above - for a table it would be:

df=spark.table('default.people')

Upvotes: 0

Anthony Mc Nicol
Anthony Mc Nicol

Reputation: 1

  1. Navigate to "Data" -> "Create Table"
  2. Drop the people.json file onto the "Files" pane (Like this)
  3. Don't click on the button "Create Table with UI" but just copy the path that is automatically created below the file. E.g. "/FileStore/tables/people.json"
  4. Now you can paste the correct path and use the read.json command. It will look something like this: df = spark.read.json('/FileStore/tables/people.json')

Upvotes: 0

Steven
Steven

Reputation: 15283

why do you want to use spark.read.json? In you picture, "people" is clearly a table.

If you want to read a table in spark, you do : spark.table("people").

You use spark.read.json when you want to read a json file. In that case you do : spark.read.json("absolute/path/to/json/file") but you need to know that path (and that's exactly the error you've got). I have no idea where your files are stored, that's entirely on you to know it. Where did you put them ?

Upvotes: 2

Related Questions