How to create log file (path sohuld be ADLS) in databricks pyspark

Question

I have written a pyspark job in databricks. My objective is to save information, exception and errors into log file. This log file should be present in ADLS.

Can someone provide me guidance on this.

Answer 1

You can set logs to be sent to a DBFS location by specifying it in the advanced settings of the cluster details page.

In the cluster page, click Edit and expand the Advanced Options. There's a Logging tab where you can input where you want the logs to go. Databricks will create a folder in this location based off the cluster ID.