CODEWITHSUNDEEP
hadoopconfigurationhdfshbase
user3091996
user3091996

Reputation:

Is there a maximum storage space configuration in HDFS or HBase?

Is there a maximum storage space configuration in HDFS or HBase?

I've found

in the hadoop documentation

For HBase I've found some heap size configurations and compaction intervals, Memstore Flush Size, but none of these seem to regulate maximum size for a single node.

Is there any configuration for either HBase or HDFS that regulates how much space they will occupy in a single node?

(I am running tests on a single machine)

Upvotes: 1

Views: 393

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191701

Generally speaking, dfs.data.dir will be a formatted volume that is mounted specifically for HDFS data. Therefore, the "maximum" is the number of physical SATA/USB/M.2 NVME connectors on the datanode's motherboard times the size of the largest hard-drives you can find.

If not using dedicated volumes/devices, the max is still limited by the disk sizes, but dfs.datanode.du.reserved will leave space for the OS to run on its own.

Neither is related to memory usage / heap space

Upvotes: 0

Related Questions