CODEWITHSUNDEEP
python-3.xxmlapache-sparkpysparkazure-databricks
Pradeep Kaja
Pradeep Kaja

Reputation: 125

Failed to save a file in azure data lake from azure data bricks

I'm trying to save the string content into azure data lake as XML content.

a string variable contains below mentioned xml content.

<project>
    <dateformat>dd-MM-yy</dateformat>
    <timeformat>HH:mm</timeformat>
    <useCDATA>true</useCDATA>
</project>

i have used the below code to process the file into data lake.

xmlfilewrite = "/mnt/adls/ProjectDataDecoded.xml"
with open(xmlfilewrite , "w") as f:
    f.write(project_processed_var)

it throws the following error: No such file or directory: '/mnt/adls/ProjectDataDecoded.xml"

I'm able to access the data lake by using the above mounting point but unable do with the above function "open".

can anyone help me?

Upvotes: 1

Views: 681

Answers (2)

Pradeep Kaja
Pradeep Kaja

Reputation: 125

Issue is solved.

In databricks when you have a mount point existing on Azure Data Lake,we need to add "/dbfs" to the path and pass it to OPEN function. The issue is solved by using below code

xmlfilewrite = "/dbfs/mnt/adls/ProjectDataDecoded.xml"
with open(xmlfilewrite , "w") as f:
    f.write(project_processed_var)

Upvotes: 3

SA2010
SA2010

Reputation: 323

You could try using the Spark-XML library. Convert your string to a dataframe where each row denotes one project. Then you can write it to ADLS in this way.

df.select("dateformat", "timeformat","useCDATA").write \
  .format('xml') \
  .options(rowTag='project', rootTag='project') \
  .save('/mnt/adls/ProjectDataDecoded.xml')

Here is how you can include an external library -https://docs.databricks.com/libraries.html#create-a-library

Upvotes: 0

Related Questions