Using Azure Function APP in Data Factory to run Python script

Question

I am merging to CSV files from a blob storage and uploading it to a Data Lake storage(Gen 2). The code works in PyCharm and VS Code, but I would like to run it in an Azure Data Factory pipeline using a function app. I get this error if I try to run it in a pipeline: "Operation on target Azure Function1 failed: Call to provided Azure function 'name' failed with status-'Unauthorized' and message - 'Invoking Azure function failed with HttpStatusCode - Unauthorized.'."

import azure.functions as func
import pandas as pd
import logging
from azure.storage.blob import BlobServiceClient
from azure.storage.filedatalake import DataLakeServiceClient

def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    STORAGEACCOUNTURL= 'https://storage.blob.core.windows.net/'
    STORAGEACCOUNTKEY= '****'
    LOCALFILENAME= ['file1.csv', 'file2.csv']
    CONTAINERNAME= 'inputblob'

    file1 = pd.DataFrame()
    file2 = pd.DataFrame(])
    #download from blob

    blob_service_client_instance = BlobServiceClient(account_url=STORAGEACCOUNTURL, credential=STORAGEACCOUNTKEY)

    for i in LOCALFILENAME:
        with open(i, "wb") as my_blobs:
            blob_client_instance = blob_service_client_instance.get_blob_client(container=CONTAINERNAME, blob=i, snapshot=None)
            blob_data = blob_client_instance.download_blob()
            blob_data.readinto(my_blobs)
            if i == 'file1.csv':
                file1 = pd.read_csv(i)
            if i == 'file2.csv':
                file2 = pd.read_csv(i)

    # load

    # join the 2 dataframes into the final dataframe
    summary = pd.merge(left=file1, right=file2, on='key', how='inner')
        
    summary.to_csv(path_or_buf=r'path\summary.csv', index=False, encoding='utf-8')

    global service_client
            
    service_client = DataLakeServiceClient(account_url="https://storage.dfs.core.windows.net/", credential='****')
        
    file_system_client = service_client.get_file_system_client(file_system="outputdatalake")

    directory_client = file_system_client.get_directory_client("functionapp") 

    file_client = directory_client.create_file("merged.csv")
            
    local_file = open(r"path\summary.csv",'rb') 

    file_contents = local_file.read()

    file_client.upload_data(file_contents, overwrite=True) 

    return func.HttpResponse("This HTTP triggered function executed successfully.")

Answer 1

I tried to reproduce with python based http trigger and after first deployement faced the below error

Call to provided Azure function 'HttpTriggerT' failed with status-'Unauthorized' and message - 'Invoking Azure function failed with HttpStatusCode - Unauthorized.'.

Note: Try to refresh and restart the Function App service after deployment or changes to it. This is know to solve few transient issues.

Ideally you can use function key (for particular single function) or master/host key (for all functions in the function app service) to allow access. Managed identity provides secure access to the entire function app

1. For using function key,

Navigate to your function app > functions > your_function > Function Keys

Copy the key and add in the functions linked service to authorize

2. For using Managed Identity

Further I have made following changes to get it working.

Navigate to your deployed function app, Settings > Identity > Turn on System assigned managed identity.

Add an identity provider. Settings > Authentication > Microsoft Identity

Create Managed Identity for ADF:

Add credentials to ADF

Finally edit Azure functions linked service

Get the resource ID from the AAD App registered as identity provider

Test function call in pipeline