VivekAnandChakravarthy
Reputation: 699
Bad Request - Null Message - Error null from the ADF - Copy data Pipeline
Continuity to one of my previous question:
I have created the ADF Pipeline for copying the blob containers and its data inside between the storage accounts as shown below:
For-each activity settings:
Pipeline parameter is container_name
and its value is array type:
[
{
"source_ctr":"app-backup",
"target_ctr":"app-backup"
},
{
"source_ctr":"azure-webjobs-hosts",
"target_ctr":"azure-webjobs-hosts"
},
{
"source_ctr":"azure-webjobs-secrets",
"target_ctr":"azure-webjobs-secrets"
},
{
"source_ctr":"data-app",
"target_ctr":"data-app"
},
{
"source_ctr":"data-apptemp",
"target_ctr":"data-apptemp"
},
{
"source_ctr":"data-uat",
"target_ctr":"data-dev"
},
{
"source_ctr":"db-backup",
"target_ctr":"db-backup"
}
]
This is the 1st Copy Data activity - Source configuration for copying the containers between the storage accounts:
CopyData1 - Sink Configuration:
Copy Data 2 - Source Configuration:
Sink Configuration:
Created two data sets & linked services for source and destination storage account connections - Test Connection is successful.
When I run the pipeline, I'm getting this error:
{
"code": "BadRequest",
"message": null,
"target": "pipeline//runid/<alpha-numeric-value>",
"details": null,
"error": null
}
Pipeline in Json:
{
"name": "AzBlobCtrDataCopyPipeline",
"properties": {
"activities": [
{
"name": "ForEach",
"type": "ForEach",
"dependsOn": [],
"typeProperties": {
"items": {
"value": "@pipeline().parameters.container_name",
"type": "Expression"
},
"activities": [
{
"name": "Copy data1",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "BinarySource",
"storeSettings": {
"type": "AzureBlobStorageReadSettings",
"recursive": true,
"wildcardFolderPath": "*",
"deleteFilesAfterCompletion": false
},
"formatSettings": {
"type": "BinaryReadSettings"
}
},
"sink": {
"type": "BinarySink",
"storeSettings": {
"type": "AzureBlobStorageWriteSettings",
"copyBehavior": "PreserveHierarchy"
}
},
"enableStaging": false
},
"inputs": [
{
"referenceName": "SrcSubscnBlobstorageds",
"type": "DatasetReference",
"parameters": {
"container_name": {
"value": "@item().source_ctr",
"type": "Expression"
}
}
}
],
"outputs": [
{
"referenceName": "DestnSubsnBlobstorageds",
"type": "DatasetReference",
"parameters": {
"container_name": {
"value": "@item().target_ctr",
"type": "Expression"
}
}
}
]
},
{
"name": "Copy data2",
"type": "Copy",
"dependsOn": [
{
"activity": "Copy data1",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "BinarySource",
"storeSettings": {
"type": "AzureBlobStorageReadSettings",
"recursive": false,
"wildcardFileName": "*",
"deleteFilesAfterCompletion": false
},
"formatSettings": {
"type": "BinaryReadSettings"
}
},
"sink": {
"type": "BinarySink",
"storeSettings": {
"type": "AzureBlobStorageWriteSettings"
}
},
"enableStaging": false
},
"inputs": [
{
"referenceName": "SrcSubscnBlobstorageds",
"type": "DatasetReference",
"parameters": {
"container_name": {
"value": "@item().source_ctr",
"type": "Expression"
}
}
}
],
"outputs": [
{
"referenceName": "DestnSubsnBlobstorageds",
"type": "DatasetReference",
"parameters": {
"container_name": {
"value": "@item().target_ctr",
"type": "Expression"
}
}
}
]
}
]
}
}
],
"parameters": {
"sourceSAS": {
"type": "string",
"defaultValue": "?hiddenSAStoken"
},
"destinationSAS": {
"type": "string",
"defaultValue": "?hiddenSAStoken"
},
"container_name": {
"type": "array",
"defaultValue": [
{
"source_ctr": "app-backup",
"target_ctr": "app-backup"
},
{
"source_ctr": "azure-webjobs-hosts",
"target_ctr": "azure-webjobs-hosts"
},
{
"source_ctr": "azure-webjobs-secrets",
"target_ctr": "azure-webjobs-secrets"
},
{
"source_ctr": "data-app",
"target_ctr": "data-app"
},
{
"source_ctr": "data-apptemp",
"target_ctr": "data-apptemp"
},
{
"source_ctr": "data-uat",
"target_ctr": "data-dev"
},
{
"source_ctr": "db-backup",
"target_ctr": "db-backup"
},
{
"source_ctr": "taskhubdev-applease",
"target_ctr": "taskhubdev-applease"
},
{
"source_ctr": "taskhubdev-leases",
"target_ctr": "taskhubdev-leases"
},
{
"source_ctr": "taskhubuat-applease",
"target_ctr": "taskhubdev-applease"
},
{
"source_ctr": "taskhubuat-largemessages",
"target_ctr": "taskhubdev-largemessages"
},
{
"source_ctr": "taskhubuat-leases",
"target_ctr": "taskhubdev-leases"
},
{
"source_ctr": "taskhubuatlocal-applease",
"target_ctr": "taskhubdevlocal-applease"
},
{
"source_ctr": "taskhubuatlocal-largemessages",
"target_ctr": "taskhubdevlocal-largemessages"
},
{
"source_ctr": "taskhubuatlocal-leases",
"target_ctr": "taskhubdevlocal-leases"
},
{
"source_ctr": "taskhubuatlocal2-applease",
"target_ctr": "taskhubdevlocal2-applease"
},
{
"source_ctr": "taskhubuatlocal2-largemessages",
"target_ctr": "taskhubdevlocal2-largemessages"
},
{
"source_ctr": "taskhubuatlocal2-leases",
"target_ctr": "taskhubdevlocal2-leases"
},
{
"source_ctr": "taskhubuatlocaln-applease",
"target_ctr": "taskhubdevlocaln-applease"
},
{
"source_ctr": "taskhubuatlocaln-leases",
"target_ctr": "taskhubdevlocaln-leases"
},
{
"source_ctr": "taskhubuatlocalnn-applease",
"target_ctr": "taskhubdevlocalnn-applease"
},
{
"source_ctr": "taskhubuatlocalnn-largemessages",
"target_ctr": "taskhubdevlocalnn-largemessages"
},
{
"source_ctr": "taskhubuatlocalnn-leases",
"target_ctr": "taskhubdevlocalnn-leases"
},
{
"source_ctr": "taskhubuatnn-applease",
"target_ctr": "taskhubdevnn-applease"
},
{
"source_ctr": "taskhubuatnn-largemessages",
"target_ctr": "taskhubdevnn-largemessages"
},
{
"source_ctr": "taskhubuatnn-leases",
"target_ctr": "taskhubdevnn-leases"
},
{
"source_ctr": "taskhubuatnnlocal-applease",
"target_ctr": "taskhubdevnnlocal-applease"
},
{
"source_ctr": "taskhubuatnnlocal-largemessages",
"target_ctr": "taskhubdevnnlocal-largemessages"
},
{
"source_ctr": "taskhubuatnnlocal-leases",
"target_ctr": "taskhubdevnnlocal-leases"
}
]
}
},
"annotations": [],
"lastPublishTime": "2024-04-22T00:25:15Z"
},
"type": "Microsoft.DataFactory/factories/pipelines"
}
Can anyone help me here what and where i'm doing the mistake.
Upvotes: 1
Views: 340
Answers (1)
Bhavani
Reputation: 5317
When I was defining the pipeline parameter, I missed the '@', which resulted in the error shown above. After defining the parameter as @pipeline().parameters.container_name, my pipeline ran successfully without any errors.
- Ensure that you have defined parameters with the correct syntax.
- Check that there are no syntax errors in the pipeline.
- Configure the pipeline correctly.
For more information, you can refer to the MS Q&A.
Upvotes: 1
Related Questions
- Cannot insert the value NULL into column -ADF Pipeline failure
- ADF Pipeline Copy data "BadRequest" error - SQL to Parquet using expression builder for Mapping
- Copy data activity in ADF is not deleting data in Azure Table Storage
- "Bad Request" message from Import Schema of Copy Activity in Azure Data Factory (ADF)
- adf v2 copy data tool with 787 tables (sql server) only creates pipeline with 340 in pipeline parameters json. why?
- Azure data factory copy activity from Storage to SQL: hangs at 70000 rows
- ADF Copy Data activity - reference Source column from dynamic expression for Sink stored procedure parameter value
- Running data flows in Azure Data Factory 4 times slower than running in Azure SSIS data flow