CODEWITHSUNDEEP
scalaazuresqoopazure-data-factoryazure-data-lake
ASH
ASH

Reputation: 20302

Having issues exporting from data lake to Azure SLQ Server DB

I'm trying to use Sqoop to export data from my data lake to a table that I setup in Azure SQL Server. This is all in Azure Data Factory. The schema of the table matches the schema of the file name, with one exception...I have a column named 'file_name' in the table, which I want to populate from the file itself (this field is not in the raw data). Here is the code that I tried.

sqoop-export 
--connect "jdbc:sqlserver://server_name.database.windows.net:1433;username=user_name;password='pass word';database=db_name" 
--table REUTERS_CW 
--export-dir adl://rawdata/corp/file_name.gz 
--input-fields-terminated-by "|"

When I run that, I get this error:

<console>:10: error: identifier expected but string literal found.
--input-fields-terminated-by "|"

There are three things that I'm not sure about.

#1) My password actually has spaces in it; I think wrapping that in quotes will fix it.

#2) I'm running this in Scala in Azure Databricks; I'm guessing that's oen way to do it, but I don't know for sure.

#3) I'm not certain about how to copy the data from the file and simultaneously append the file name relative to the data that's supposed to be copied over.

I'm following the example from the link below.

https://learn.microsoft.com/en-us/azure/data-lake-store/data-lake-store-data-transfer-sql-sqoop

Upvotes: 1

Views: 77

Answers (1)

Leon Yue
Leon Yue

Reputation: 16401

What I get from sqoop-export document is that it seems not support compressed files.

You can reference sqoop-export Syntax.

sqoop-export:

enter image description here The --export-dir argument and one of --table or --call are required. These specify the table to populate in the database (or the stored procedure to call), and the directory in HDFS that contains the source data.

sqoop-import:

enter image description here

Hope this helps.

Upvotes: 2

Related Questions