CODEWITHSUNDEEP
azure-data-factoryparquetdata-warehousemicrosoft-fabric
SRP
SRP

Reputation: 1173

Getting Copying from Column may have data truncation warning during mapping and type Float not compatible error in Data Factory under MS Fabric?

I am trying to create a pipeline in Data Factory service present under Microsoft Fabric in its trail license.

This file is supposed to copy the data present in a file which is in *.txt.gz format present at my azure blob container to my data warehouse present in MS Fabric.

However, while mapping the source and destination column I get the warning as shown below for one of the column -

enter image description here

When i proceed with executing the pipeline it fails and gives below error. Post updating the column type from float to varchar(8000) and Datatime2 to varchar(8000) it works fine and ingestion gets successful.

"ErrorCode=UserErrorSqlDWCopyCommandError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=SQL DW Copy Command operation failed with error 'Column 'PACKAGE_SIZE' of type 'FLOAT' is not compatible with external data type 'Parquet physical type: BYTE_ARRAY, logical type: UTF8', please try with 'VARCHAR(8000)'."

enter image description here

I don't understand both the datatype are supported in Fabric as per this link

https://learn.microsoft.com/en-us/fabric/data-warehouse/data-types

Then why my pipeline fails, am i missing any configuration or mis-interpretating the error message?

Upvotes: 0

Views: 430

Answers (1)

Bhavani
Bhavani

Reputation: 5317

enter image description here

The error you're encountering indicates a type mismatch between the data in your Parquet file and the target column in your SQL Data Warehouse. Specifically, the Parquet file has data stored as a BYTE_ARRAY with a UTF8 logical type, which corresponds to a string type in SQL, but your target column PACKAGE_SIZE is defined as a FLOAT. So, according to the error compatible data type of BYTE_ARRAY with a UTF8 logical type is VARCHAR(8000) .

That may be reason to get success of pipeline when changing column type from float to varchar(8000) and Datatime2 to varchar(8000). For more information you can refer to this.

According to the MS document .gz extension used to read parquet file. Thay may be the reason for when data file is in text format compressed as .gz then it is considering it as parquet file format.

Upvotes: 0

Related Questions