CODEWITHSUNDEEP
apache-sparkapache-spark-sqldatabricksazure-databrickspyspark
KurryF
KurryF

Reputation: 81

How to read/load local files in Databricks?

is there anyway of reading files located in my local machine other than navigating to 'Data'> 'Add Data' on Databricks.

in my past experience using Databricks, when using s3 buckets, I was able to just read and load a dataframe by just specifying the path like so: i.e

df = spark.read.format('delta').load('<path>')

is there any way i can do something like this using databricks to read local files?

Upvotes: 3

Views: 27142

Answers (1)

Raphael K
Raphael K

Reputation: 2353

If you use the Databricks Connect client library you can read local files into memory on a remote Databricks Spark cluster. See details here.

The alternative is to use the Databricks CLI (or REST API) and push local data to a location on DBFS, where it can be read into Spark from within a Databricks notebook. A similar idea would be to use the AWS CLI to put local data into an S3 bucket that can be accessed from Databricks.

It sounds like what you are looking for is Databricks Connect, which works with many popular IDEs.

Upvotes: 1

Related Questions