CODEWITHSUNDEEP
pythonjsondatabricksazure-data-lake
mizzlosis
mizzlosis

Reputation: 587

Save dict as json using python in databricks

I am trying to save a dict as json in azure data lake/databricks however I am getting a File not found error. Any clue what I am doing wrong?

import json

test_config = {
  "expectations": [
    {
      "kwargs": {
        "column": "role",
        "value_set": [
          "BBV",
          "GEM"
        ]
      },
      "expectation_type": "expect_column_distinct_values_to_be_in_set",
      "meta": {}
    }]
    }

path = '/mnt/lake/enriched/checks/'
json_file = 'test_idm_expectations.json'

with open(path+json_file, "w") as fp:
    json.dump(test_config , fp)

And the error I am getting is:

FileNotFoundError: [Errno 2] No such file or directory: "/mnt/lake/enriched/checks/test_idm_expectations.json"

Variations of the path with /dbfs/mnt/lake/enriched/checks/ or dbfs:mnt/lake/enriched/checks/ also do not work.

Any help would be super appreciated. Thanks!

Upvotes: 0

Views: 1971

Answers (3)

mizzlosis
mizzlosis

Reputation: 587

Thanks @Remzinho & @Rakesh, the folder structure was indeed not created. adding this snippet to my code before saving the data, solved the issue.

if not os.path.exists(path):
    os.mkdir(path)

with open(path+json_file, "w") as fp:
    json.dump(test_config , fp)

Upvotes: 0

Rakesh Govindula
Rakesh Govindula

Reputation: 11319

I have reproduced the above and got same error.

enter image description here

The above error occurs if we give wrong folder name in the path. you can see as there is no sourcefolder1 in my container, it generated that error. It will not create new folder for us. It only creates that particular file(mysample1.json in this case).

You can see if I gave correct folder(which is created before) sourcefolder the file is generated successfully.

enter image description here

Give path like /dbfs/mnt/folder/ only as I got same error when I gave path like /mnt/folder/.(open() not identifying paths in the mount might be the reason for it).

File Generated:

enter image description here

Upvotes: 1

Remzinho
Remzinho

Reputation: 549

Ensure your python environment sees the mountpoint. You can use os.path.ismount for that. Also, check if the folder tree structure exists. json.dumps will create your file, but only if the folder exists.

Also, tip: to keep indentation, use indent=2 or whatever number of spaces you want in your json, to be pretty printed.

Upvotes: 1

Related Questions