CODEWITHSUNDEEP
hadoopsqoopavro
Prabhanj
Prabhanj

Reputation: 282

How to pass column names having spaces to sqoop --map-column-java

I have to import data using sqoop, my source column names are having spaces in between them, so while I am adding it in --map-column-java parameter getting the error.

Sample Sqoop import:

sqoop import --connect jdbc-con --username "user1" --query "select * from table where \$CONDITIONS" --target-dir /target/path/ -m 1 --map-column-java data col1=String, data col2=String, data col3=String --as-avrodatafile

Column names:

data col1,
data col2,
data col3

Error:

19/03/07 07:31:55 DEBUG sqoop.Sqoop: Malformed mapping. Column mapping should be the form key=value[,key=value]* java.lang.IllegalArgumentException: Malformed mapping. Column mapping should be the form key=value[,key=value]* at org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1355) at org.apache.sqoop.SqoopOptions.setMapColumnJava(SqoopOptions.java:1375) at org.apache.sqoop.tool.BaseSqoopTool.applyCodeGenOptions(BaseSqoopTool.java:1363) at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:1011) at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435) at org.apache.sqoop.Sqoop.run(Sqoop.java:135) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) Malformed mapping. Column mapping should be the form key=value[,key=value]*

Upvotes: 1

Views: 2225

Answers (2)

Sanjay Sheel
Sanjay Sheel

Reputation: 97

Instead of using you can use this one method I have used it and it works

here I am casting the columns to string so that timestamp could not change to int

keep note of that point It will help you to make your string properly

address = <localhost/server-ip-address/>

port = put your database port number

Sqoop is expecting the comma-separated list of mapping in form 'name of column'='new type'

columns-name =  give your database column name of timestamp or date time to date


database-name = give your datbase name 

database-user-name = put your user name 

password = put your password

demo to understand the code properly

sqoop import  --map-column-java "columns-name=String"    --connect jdbc:postgresql://address:port/database-name     --username user-name     --password  database-password --query "select * from demo where \$CONDITIONS;"  -m 1 --target-dir /jdbc/star --as-parquetfile  --enclosed-by '\"'

demo of code for single-column

sqoop import  --map-column-java "date_of_birth=String"    --connect jdbc:postgresql://192.168.0.1:1928/alpha     --username postgres     --password  mysecretpass --query "select * from demo where \$CONDITIONS;"  -m 1 --target-dir /jdbc/star --as-parquetfile  --enclosed-by '\"'

demo of code for dealing with multiple columns

sqoop import  --map-column-java "date_of_birth=String,create_date=String"    --connect jdbc:postgresql://192.168.0.1:1928/alpha     --username postgres     --password  mysecretpass --query "select * from demo where \$CONDITIONS;"  -m 1 --target-dir /jdbc/star --as-parquetfile  --enclosed-by '\"'

Upvotes: 0

Prabhanj
Prabhanj

Reputation: 282

Able to resolve this issue:

1. Spaces issue: sqoop import --connect jdbc-con --username "user1" --query "select * from table where \$CONDITIONS" --target-dir /target/path/ -m 1 --map-column-java "data col1=String, data col2=String, data col3=String" --as-avrodatafile

2. ERROR tool.ImportTool: Import failed: Cannot convert SQL type 2005: 3 columns in source are having 2005 and nvarchar added them in --map-column-java resolved this issue

3. org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.UnresolvedUnionException: Not in union ["null","long"]: 1****

This is causing due to using * in select query, so modified sqoop query as:

sqoop import --connect jdbc-con --username "user1" --query "select [col1,data col2,data col3] from table where \$CONDITIONS" --target-dir /target/path/ -m 1 --map-column-java "data col1=String, data col2=String, data col3=String" --as-avrodatafile

Upvotes: 1

Related Questions