CODEWITHSUNDEEP
hadoophivehdfspartitioning
optimus_prime
optimus_prime

Reputation: 827

hive/hdfs moving data not working as expected

I had a table in hive called as test at location say 'hdfs://location1/partition='x'' and moved all the data to 'hdfs://location2/partition='x''.

hdfs dfs -mv /location1 /location2

Then I did 

alter table test set location 'hdfs://location2'.

On doing 

 hdfs dfs -ls /location2

I see all the data in the right partition

Querying to get counts i.e. 

select count(*) from test

works fine.

But doing

 select * from test

pulls no records.

Unable to figure what went wrong while moving.

Upvotes: 0

Views: 235

Answers (1)

Amit Kumar
Amit Kumar

Reputation: 1584

You need to drop the existing partitions that was pointing to the original location "hdfs://location1/partition='x'" manually. Use below command to drop all the partitions manually:

alter table test drop partition(partition='x');

Once all the partitions are dropped run the below command to update the new partitions in hive metastore:

msck repair table test;

Why this? Because since the location of table was changed but the hive metastore was not updated with the new partitions in new location. The hive metastore is still holding the information about the partitions from old location. Once you drop partition and run the

msck repair

command, the hive metastore will get updated with the new partitions from new location.

Upvotes: 1

Related Questions