CODEWITHSUNDEEP
azure-data-factory
Alex K
Alex K

Reputation: 139

ADF: Using look up and foreach activities to get and array ouput

I'm using lookup activity to scan my rows and foreach activity to concatinate my values into an array without any success.

The csv file that lookup is scanning looks like this:

value
a
b
c
d

my foreach looks like this : enter image description here

and inside my foreach I have an set variable activity which I'm aware that the datatype is not matching but do not know how to solve

enter image description here

Upvotes: 1

Views: 2193

Answers (2)

Kranthi Pakala
Kranthi Pakala

Reputation: 1424

The error you are seeing is because you are trying to pass an array value to a string type variable.

For your requirement you will have to use AppendVariable activity inside your ForEach and load all the iteration values to your AppendVariable of type array. Then outside of you ForEach, have setVariable activity of type array variable and map the AppendVariable to SetVariable

Below is sample pipeline JSON for the same requirement. You can reuse it by replacing the source dataset

{
"name": "pl_LookupToArray",
"properties": {
    "activities": [
        {
            "name": "Lookup1",
            "type": "Lookup",
            "dependsOn": [],
            "policy": {
                "timeout": "0.12:00:00",
                "retry": 0,
                "retryIntervalInSeconds": 30,
                "secureOutput": false,
                "secureInput": false
            },
            "userProperties": [],
            "typeProperties": {
                "source": {
                    "type": "DelimitedTextSource",
                    "storeSettings": {
                        "type": "AzureBlobFSReadSettings",
                        "recursive": true,
                        "enablePartitionDiscovery": false
                    },
                    "formatSettings": {
                        "type": "DelimitedTextReadSettings"
                    }
                },
                "dataset": {
                    "referenceName": "DelimitedText28",
                    "type": "DatasetReference"
                },
                "firstRowOnly": false
            }
        },
        {
            "name": "ForEach1",
            "type": "ForEach",
            "dependsOn": [
                {
                    "activity": "Lookup1",
                    "dependencyConditions": [
                        "Succeeded"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "items": {
                    "value": "@activity('Lookup1').output.value",
                    "type": "Expression"
                },
                "isSequential": true,
                "activities": [
                    {
                        "name": "Append variable1",
                        "type": "AppendVariable",
                        "dependsOn": [],
                        "userProperties": [],
                        "typeProperties": {
                            "variableName": "appendVarArray",
                            "value": {
                                "value": "@item().value",
                                "type": "Expression"
                            }
                        }
                    }
                ]
            }
        },
        {
            "name": "Set variable2",
            "type": "SetVariable",
            "dependsOn": [
                {
                    "activity": "ForEach1",
                    "dependencyConditions": [
                        "Succeeded"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "variableName": "finalArrayValue",
                "value": {
                    "value": "@variables('appendVarArray')",
                    "type": "Expression"
                }
            }
        }
    ],
    "variables": {
        "appendVarArray": {
            "type": "Array"
        },
        "finalArrayValue": {
            "type": "Array"
        }
    },
    "annotations": []
}

}

Here is how the pipeline flow looks:

enter image description here

Here is how your final output value of your array type variable looks like :

{
"name": "finalArrayValue",
"value": [
    "a",
    "b",
    "c",
    "d"
]

}

Upvotes: 3

Rakesh Govindula
Rakesh Govindula

Reputation: 11529

concatinate my values into an array without any success.

Use append variable inside ForEach activity.

My lookup activity:

enter image description here

create an array variable. The second array variable is just for showing output.

enter image description here

append variable:

@item().value

enter image description here

Output:

enter image description here

Upvotes: 0

Related Questions