CODEWITHSUNDEEP
hadoophdfs
Rio mario
Rio mario

Reputation: 283

Data retention in Hadoop HDFS

We have a Hadoop cluster with over 100TB data in HDFS. I want to delete data older than 13 weeks in certain Hive tables.

Are there any tools or way I can achieve this?

Thank you

Upvotes: 1

Views: 7583

Answers (1)

brandon.bell
brandon.bell

Reputation: 1411

To delete data older then a certain time frame, you have a few options.

First, if the Hive table is partitioned by date, you could simply DROP the partitions within Hive and remove their underlying directories.

Second option would be to run an INSERT to a new table, filtering out the old data using a datestamp (if available). This is likely not a good option since you have 100TB of data.

A third option would be to recursively list the data directories for your Hive tables. hadoop fs -lsr /path/to/hive/table. This will output a list of the files and their creation dates. You can take this output, extract the date and compare against the time frame you want to keep. If the file is older then you want to keep, run a hadoop fs -rm <file> on it.

A fourth option would be to grab a copy of the FSImage: curl --silent "http://<active namenode>:50070/getimage?getimage=1&txid=latest" -o hdfs.image Next turn it into a text file. hadoop oiv -i hdfs.image -o hdfs.txt. The text file will contain a text representation of HDFS, the same as what hadoop fs -ls ... would return.

Upvotes: 4

Related Questions