How to download a file from Databricks filesystem without using CLI?

Question

I saved a file from Databricks notebook as follows:

joblib.dump(rf, '/dbfs/FileStore/model.pkl', compress=9)

How can I download it to my local filesystem without using CLI?

I tried as follows:

https://westeurope.azuredatabricks.net/dbfs/FileStore/model.pkl?o=####

But it does not work.

Answer 1

Easiest is that you start to write to s3 bucket as

df.write.format("com.databricks.spark.csv").option("header", "true") \
  .save("s3://<your folder desired(exist or not)/file")

Now you can retrieve it with aws s3 cp <your folder desired(exist or not)/file <local file path>

Answer 2

I figured it out myself:

https://westeurope.azuredatabricks.net/files/model.pkl?o=####

More details can be found here.