CODEWITHSUNDEEP
pythonazureapache-sparkpysparkdatabricks
user16714516
user16714516

Reputation:

How to create logging from Python notebook in Databricks to ADLS Gen2 (without mounting)?

I'm trying to create a logging mechanism inside Databricks Python notebook. Tried using below code to achieve the same -

import logging

def create_logger(name,log_path=None):
  
    logger = logging.getLogger(name)  
    logger.setLevel(logging.DEBUG)
    formatter    = logging.Formatter("%(asctime)s - %(levelname)-8s - %(message)s")  
    stream_handler = logging.StreamHandler()
    stream_handler.setFormatter(formatter)
    logger.addHandler(stream_handler)
    
    if log_path is not None:        
        file_handler = logging.FileHandler(log_path)
        file_handler.setFormatter(formatter)
        logger.addHandler(file_handler)
        
    return logger

However, whenever I'm trying to call the function as below -

from datetime import date, datetime
current_date = date.today()
current_timestamp = datetime.strftime(datetime.now(),"%Y%m%d%H%M%S")

name = "temp_logs"
log_path = f"abfss://{storageContainer}@{storageAccount}.dfs.core.windows.net/{target_dir}/logs/{current_date}/{name}_{current_timestamp}.txt"

logger = create_logger(name = name,log_path = log_path)

This is giving error as -

[Errno 2] No such file or directory: /databricks/driver/abfss:/[email protected]/test/logs/2021-09-13/temp_logs_20210913101150.txt'

Is there a way to handle this (without using mount point location) ?

Upvotes: 0

Views: 7826

Answers (1)

SaiKarri-MT
SaiKarri-MT

Reputation: 1301

We can try using “BlobStorageRotatingFileHandler” by importing it as below:

from azure_storage_logging.handlers import BlobStorageRotatingFileHandler

Can refer to the python document about azure-storage-logging as it provides functionality to send output from the standard Python logging APIs to Microsoft Azure Storage.

enter image description here

Sample code as below:

import logging
from azure_storage_logging.handlers import TableStorageHandler

# configure the handler and add it to the logger
logger = logging.getLogger('example')
handler = TableStorageHandler(account_name='mystorageaccountname',
                              account_key='mystorageaccountkey',
                              extra_properties=('%(hostname)s',
                                                '%(levelname)s'))
logger.addHandler(handler)

# output log messages
logger.info('info message')
logger.warning('warning message')
logger.error('error message')

Use the above logs to record the error messages.

And for "directory not found" we need to check the path. Please have a look on the documentation

Upvotes: 1

Related Questions