Azure Databricks : Save file in Azure Datalake directory folder

Question

I would like to save file in my azure datalake gen2 directory by it is not working.

When I save the file without directory it works but with Directory "Data" it is not working.

Here is the code :

# Create a file in local data directory to upload and download
local_path = "./Data"
local_file_name = "quickstart" + str(uuid.uuid4()) + ".txt"
upload_file_path = os.path.join(local_path, local_file_name)

# Write text to the file
file = open(upload_file_path, 'w')
file.write("Hello, World!")
file.close()

# Create a blob client using the local file name as the name for the blob
blob_client = blob_service_client.get_blob_client(container=container_name, blob=local_file_name)

print("\nUploading to Azure Storage as blob:\n\t" + local_file_name)

# Upload the created file
with open(upload_file_path, "rb") as data:
    blob_client.upload_blob(data)

Here is the error:

FileNotFoundError: [Errno 2] No such file or directory: './Data/quickstart564851de-67a3-4b28-9af1-049298f26408.txt'
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<command-2012583145450168> in <module>
      5 
      6 # Write text to the file
----> 7 file = open(upload_file_path, 'w')
      8 file.write("Hello, World!")
      9 file.close()

FileNotFoundError: [Errno 2] No such file or directory: './Data/quickstart564851de-67a3-4b28-9af1-049298f26408.txt'

The folder "Data" exist in my data lake :

from azure.storage.filedatalake import FileSystemClient

file_system = FileSystemClient.from_connection_string(connect_str, file_system_name="flatdata")

paths = file_system.get_paths()

for path in paths:
    print(path.name + '\t')

output :

Data    
doto    
new.csv 
quickstart.csv  
quickstartc0afe722-3851-4f5a-a6fa-83fd97389c43.txt  
saved.csv

Here is the documentation from microsoft : click here to see the Documentation

Thanks for your help

Answer 1

Reference to the documentation. I've created a test to upload local file to a folder of my Azure data lake.
This is my local project file structure:

The file was uploaded to the folder of my Azure data lake.

This is my python code

    import os, uuid
    from azure.storage.filedatalake import DataLakeServiceClient

    # Create a file in local data directory to upload and download
    local_path = "./data"
    local_file_name = "quickstart" + str(uuid.uuid4()) + ".txt"
    upload_file_path = os.path.join(local_path, local_file_name)

    # Write text to the file
    file = open(upload_file_path, 'w')
    file.write("Hello, World!")
    file.close()
    
    #upload the file to the folder
    file_system_client = service_client.get_file_system_client(file_system="test10")

    directory_client = file_system_client.get_directory_client("data")

    file_client = directory_client.create_file(local_file_name)

    local_file = open(upload_file_path, 'rb')

    file_contents = local_file.read()

    file_client.append_data(data=file_contents, offset=0, length=len(file_contents))

    file_client.flush_data(len(file_contents))