CODEWITHSUNDEEP
azure-data-factory
Randy Minder
Randy Minder

Reputation: 48392

Column defined in source Dataset could not be found in the actual source

I have an ADF Copy Data flow and I'm getting the following error at runtime:

enter image description here

My source is defined as follows:

enter image description here

In my data set, the column is defined as shown below:

enter image description here

As you can see from the second image, the column IsLiftStation is defined in the source. Any idea why ADF cannot find the column?

Upvotes: 2

Views: 8661

Answers (4)

John Joseph
John Joseph

Reputation: 1175

Same error for me and none of the other solutions above worked. What finally worked for me was a simple workaround they may or may not work for you: I changed the source data setting "First row as header" to False, then went back to Mapping in the pipeline and mapped the column indexes to the target fields.

This of course requires that I now deal with the actual header row ending up in my target data table. Not ideal but this did enable me to progress past this error.

Upvotes: 0

Jeremy Green
Jeremy Green

Reputation: 1

The issue is caused by an incomplete schema in one of the data sources. My solution is:

  1. Step through the data flow selecting the first schema, Import projection
  2. Go to the flow and Data Preview
  3. Repeat for each step.

In my case, there were trailing commas in one of the CSV files. This caused automated column names to be created in the import allowing me to fix the data file.

Upvotes: 0

tashb
tashb

Reputation: 61

I've had the same error. You can solve this by either selecting all columns (*) in the source and then mapping those you want to the sink schema, or by 'clearing' the mapping in which case the ADF Copy component will auto map to columns in the sink schema (best if columns have the same names in source and sink). Either of these approaches works.

Unfortunately, clicking the import schema button in the mapping tab doesn't work. It does produce the correct column mappings based on the columns in the source query but I still get the original error 'the column could not be located in the actual source' after doing this mapping.

Upvotes: 2

Li Liu
Li Liu

Reputation: 61

could you check that is there a column named 'ae_type_id' in your schema? If that's the case, could you remove that column and try again? The columns in the schema must be aligned with columns in the query.

Upvotes: 1

Related Questions