CODEWITHSUNDEEP
mysqlhivespecial-characterssqoop
User12345
User12345

Reputation: 5480

Sqoop Import replace special characters of mysql

I have 1000 tables with more than 100000 records in each table in mysql. The tables have 300-500 columns.

Some of tables have columns with special characters like .(dot) and space in the column names.

Now I want to do sqoop import and create a hive table in HDFS in a single shot query like below

sqoop import --connect ${domain}:${port}/$(database) --username ${username} --password ${password}\
--table $(table) -m 1 --hive-import --hive-database ${hivedatabase} --hive-table $(table) --create-hive-table\
--target-dir  /user/hive/warehouse/${hivedatabase}.db/$(table)

After this the hive table is created but when I query the table it shows error as

This error output is a sample output.

Error while compiling statement: FAILED: RuntimeException java.lang.RuntimeException: cannot find field emp from [0:emp.id, 1:emp.name, 2:emp.salary, 3:emp.dno]

How can we replace the .(dot) with _(underscore) while doing sqoop import itself. I would like to do this dynamically.

Upvotes: 1

Views: 2950

Answers (1)

sandeep rawat
sandeep rawat

Reputation: 4957

Use sqoop import \ with --query option rather than --table and in query use replace function .

ie

sqoop import --connect ${domain}:${port}/$(database) --username ${username} --password ${password}\ -- query 'Select col1 ,replace(col2 ,'.','_') as col from table.

Or (not recommended) write a shell script which can do find and replace "." to "_" (Grep command)at /user/hive/warehouse/${hivedatabase}.db/$(table)

Upvotes: 2

Related Questions