Unable to ingest data into Hive DB using Azure Data Factory with ODBC on Windows VM

Question

I want to ingest CSV files to a Hive DB using a Data Factory pipeline. There is Hive connector available but it can only be used as a source and not as a sink. So as far as I know the only way to ingest data is using the ODBC connector in Data Factory.

My setup looks like this: I'm using Azure Data Factory (V2) and an Azure Windows VM for the self-hosted Integration Runtime. I have set up the ODBC connector with the needed connection details for the Hive DB (the connection is working successfully). Also the ODBC connector will find any existing tables.

But when I trigger the pipeline I get following error:

Failure happened on 'Sink' side. ErrorCode=SystemErrorOdbcWriteError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR [07009] [Microsoft][ODBC] (10670) Invalid descriptor index, descriptor record does not exist, or descriptor record was not properly initialized.,Source=Microsoft.DataTransfer.ClientLibrary.Odbc.OdbcConnector,''Type=Microsoft.DataTransfer.ClientLibrary.Odbc.Exceptions.OdbcException,Message=ERROR [07009] [Microsoft][ODBC] (10670) Invalid descriptor index, descriptor record does not exist, or descriptor record was not properly initialized.,Source=Microsoft Hive ODBC Driver,'

Does anyone have an idea what is going wrong here?

Cheers, Niklas

Answer 1

Problem solved: Ingest data into Hive tables via ODBC is not supported and also shouldn't be the way to go. Use Hive activity in ADF instead which will run a HiveQL script which will ingest the data.