CODEWITHSUNDEEP
mysqlhadoophdfsbigdatasqoop
Prabhakar Yadav
Prabhakar Yadav

Reputation: 1

How to incrementally update a table from SQL to HDFS

I have a table imported to HDFS from RDBMS using Sqoop. After I make updates and inserts on RDBMS, how do I update the HDFS version, only transferring new and updated records? There is no timestamp or date column in this table. Can I use last modified incremental append?

Upvotes: 0

Views: 59

Answers (1)

Shah
Shah

Reputation: 130

Here last modified too required integer, timestamp,date value so last modified is not possible.

I think you can use one more column as index in mysql. And then you can import and incremental append.

MySQL index column formula:

CREATE INDEX index_name ON table_name (column_name);

This column name use as you want .it will be accepted duplicate data in row too.so don't worry.

Upvotes: 1

Related Questions