CODEWITHSUNDEEP
hivesqoop
Rahul Patidar
Rahul Patidar

Reputation: 209

Handling Multiple Delimiter while loading data in hive

I'm importing data from db2 using sqoop and using '|' as field delimiter.Then loading this data into hive table using load data command. In hive table create statement also I am using field delimiter as '|' but from source side for one column, out of 1M records in 30 Records I found '|' in data itself. So while loading data hive is considering it as new filed and loading into next column. Is there any way to handle this situation? I found some enclosure we can use but in data I am also getting multiple special charcter(-,/..etc) so I am not able to decide how should and where(sqoop or hive/both) I use this enclosure. Can someone please guide me how can I handle this situation. Also I can use different field delimiter (,| etc) as I m not sure about special character in data when I checked at source I found multiple characters.

sqoop import Properties:  -fields-terminated-by '|' --hive-drop-import-delims
Hive Table Properties: row format delimited fields terminated by '|' stored as textfile;
Sample Data:H No. 100,200,300,400, KbcColony-||, D Delhi

Upvotes: 0

Views: 207

Answers (1)

Soumendra Mishra
Soumendra Mishra

Reputation: 3663

You can use escaped-by option:

--escaped-by \\ \
--fields-terminated-by '|' \

Upvotes: 1

Related Questions