Flavi
Reputation: 353
How to execute Spark code locally with databricks-connect?
Is there a way to execute Spark code locally with databricks-connect?
The reason is that I would like to execute some tests as part of my CI/CD pipeline without the need to have a cluster up and running.
Upvotes: 8
Views: 2667
Answers (1)
simon_dmorias
Reputation: 2473
No, databricks-connect requires a running cluster.
If you do not use any databricks specific code (like dbutils) you can run spark locally and execute against that - assuming you can still access the data sources you need.
Upvotes: 5
Related Questions
- Databricks Connect: can't connect to remote cluster on azure, command: 'databricks-connect test' stops
- Spark Submit error when running a JAR from Azure Databricks
- Running exported databricks notebook in a local spark on a vm
- Unable to call a function in Apache Spark with Databricks
- Error Connecting to Databricks from local machine
- How to run .Net spark jobs on Databricks from Azure Data Factory?
- Create Spark context from Python in order to run databricks sql
- How to configure a custom Spark Plugin in Databricks?
- How to run a non-spark code on databricks cluster?
- Run a Spark SQL query on Databricks from .Net UI