CODEWITHSUNDEEP
pythonazurepysparkdatabricksazure-databricks
anaghapramesh
anaghapramesh

Reputation: 87

How can I connect to an on-prem windows server from azure databricks notebook using python?

Connection need to be established between azure databricks and an on-prem windows server. I tried the below python code:

import os

filePath = "\\\\SERVER001\\folder\\"
fileExtension = ".xml"

def get_file_count(filePath, fileExtension):
  try:
    fileCount = len([name for name in os.listdir(filePath) if name.endswith(fileExtension)])
    print(fileCount)
  except Exception as e:
    print(str(e))
get_file_count(filePath, fileExtension)

but it gave me the error:

[Errno 2] No such file or directory: '\\\\SERVER001\\folder\\'

It is searching within the databricks directories I guess. The connection itself is not happening. I am a beginner in databricks domain. Any help will be appreciated.

Upvotes: 1

Views: 1824

Answers (1)

Alex Ott
Alex Ott

Reputation: 87069

It's not possible out of the box, because that server is on-premise, and Databricks is in the cloud, without any knowledge about your on-premise environment.

You have two choices:

  1. You need to upload files onto DBFS, and then access them. You can do it for example via UI - via DBFS file browser (docs) or via Upload Data UI (docs). If you have a lot of files is huge, then you can use something like az-copy to upload file(s) to Azure Storage

  2. Theoretically you can setup your network environment to connect to on-premise via VPN (you need the workspace with "Bring your own VNet"), and then access the file share, but that could be challenging as you need to make sure that you have all necessary ports opened on firewalls, etc.

I would recommend to go with first option.

Upvotes: 1

Related Questions