CODEWITHSUNDEEP
azureazure-synapseazure-data-factory
Pankaj Singh
Pankaj Singh

Reputation: 29

How to trigger Synapse Analytics Workspace pipeline using C# or any other language?

Is there any way through which we can trigger the Azure Synapse Analytics Pipeline (built-in Azure Data Factory) using C# or any language?

Using the following URL and code I am able to trigger general (not part of Synapse) Azure Data Factory successfully. But when I call the same method I am not sure what will go under name of Data Factory (property: dataFactoryName)? I tried giving workspace name but it does not work.

Built-in ADF can be triggered using Blob trigger but thing is that I have many parameters and those cannot be passed through files stored in Blob.

URL: https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers

Code: client.Pipelines.CreateRunWithHttpMessagesAsync(resourceGroup, dataFactoryName, pipelineName, parameters)

Upvotes: 0

Views: 3251

Answers (3)

Muggin
Muggin

Reputation: 1291

    public async Task<string> ExecuteSynapsePipeline(IDictionary<string, object?> parameters, string? synapseEndPoint = null, string? pipeline = null)
    {
        synapseEndPoint ??= _Settings.SynapseExtractEndpoint;
        pipeline ??= _Settings.SynapseExtractPipeline;

        _Logger.LogInformation($"!!Execute synapse pipeline called with parameters: {parameters}");

        Response<CreateRunResponse?>? result = null;

        try
        {
            PipelineClient client = new(new Uri(synapseEndPoint), new DefaultAzureCredential());

            _Logger.LogInformation($"!!Pipeline {client} client resolved");

            result = await client.CreatePipelineRunAsync(
                pipeline,
                parameters: parameters);

            _Logger.LogInformation($"!!{pipeline} {result?.Value?.RunId ?? "Failed to get RunId"} called");
        }
        catch(Exception e)
        {
            string errorToLog = $"{e.Message} \n";
            if(e.InnerException is not null && !string.IsNullOrEmpty(e.InnerException.Message))
            {
                errorToLog += e.InnerException.Message;
            }
            _Errors.LogError(options.OrchestratorId, errorToLog);
            _Logger.LogError(errorToLog);
        }

        return result?.Value?.RunId ?? "";
    }

Upvotes: 0

Pankaj Singh
Pankaj Singh

Reputation: 29

I have taken guidance from the following URL

https://techcommunity.microsoft.com/t5/azure-synapse-analytics/how-to-start-synapse-pipeline-from-rest-api/ba-p/1684836

and created a C# version of it here - https://github.com/pankajsingh23/TriggerSynapsePipeline

Upvotes: 0

Joel Cochran
Joel Cochran

Reputation: 7748

According to the ADF team, there is a different SDK for Synapse Analytics. I'm in the same position, but haven't had a chance to generate a code sample yet.

It looks like you'll need the PipelineClient class to create a run, and the PipelineRunClient class to monitor it.

If you get this working, please post the sample code for future searchers.

Upvotes: 0

Related Questions