CODEWITHSUNDEEP
databricksminicondadatabricks-connect
madisonsus
madisonsus

Reputation: 11

Can I have more than one connection in databricks-connect?

I have setup on my PC a miniconda python environment where I have installed the databricks-connect package and configured the tool with databricks-connect configure to connect to a databricks instance I want to use when developing code in the US.

I have a need to connect to a different a different databricks instance for developing code in the EU and I thought I could do this by setting up a different miniconda environment and installing databricks-connect in that environment and setting the configuration in that environment to point to the new databricks instance.

Alas, this did not work. When I look at databricks-connect configure in either miniconda environment, I see the same configuration in both which is the configuration I last configured.

My question therefore is: Is there a way to have multiple databricks-connect connections at the same time and toggle between the two without having to reconfigure each time?

Thank you for your time.

Upvotes: 1

Views: 1296

Answers (1)

Alex Ott
Alex Ott

Reputation: 87224

Right now, databricks-connect relies on the central configuration file, and this causes problems. There are two approaches to workaround that:

  1. Use environment variables as described in the documentation, but they should be set somehow, plus you need to have different python environments for different versions of databricks-connect
  2. Specify parameters as spark configuration (see in the same documentation)
  3. For each DB cluster, do following:
  • separate python environment with name <name> & activate it
  • install databricks-connect into it
  • configure databricks-connect
  • move ~/.databricks-connect into ~/.databricks-connect-<name>
  • write wrapper script, that will activate python environment & symlink ~/.databricks-connect-<name> into ~/.databricks-connect (I have such script for Zsh, it could be too long to paste it here.)

Upvotes: 2

Related Questions