CODEWITHSUNDEEP
pythondistributed-computingh2o
user13867283
user13867283

Reputation:

How to connect two computers to an H2O cluster in Python

I have two computers that I want to connect to a single H2O cluster. One has 4 cores, the other has 6. When I run

h2o.init(ip = '10.0.0.89', port = 54321)

on the first computer, I get the following output:

H2O cluster uptime: 1 minutes 56 seconds 846 milliseconds
H2O cluster version:    3.8.2.3
H2O cluster name:   H2O_started_from_python_samerens_dii030
H2O cluster total nodes:    1
H2O cluster total free memory:  3.4 GB
H2O cluster total cores:    4
H2O cluster allowed cores:  4
H2O cluster healthy:    True
H2O Connection ip:  10.0.0.89
H2O Connection port:    54321
H2O Connection proxy:   None
Python Version: 3.7.4

When I run the same command on the second computer, I get the same output (except for the uptime of course). Shouldn't the total nodes have increased to 2 and the total cores have increased to 10? Am doing something wrong?

Upvotes: 1

Views: 265

Answers (1)

Erin LeDell
Erin LeDell

Reputation: 8819

What you're currently doing is starting the H2O cluster on a single machine and then connecting to it from the other machine (this is the scenario when you want two users to have access to the same H2O cluster to share data/models).

What you're trying to do is start a multi-node H2O cluster. It's not clear whether this will speed up your training or not because there's communication overhead when you use a multi-node cluster, so it's always good to check. If you are adding a larger number of cores, e.g. 40 + 40 for a total of 80 cores, I would expect that to speed things up in most cases, but adding 4 to 6 might not help that much (I just want to emphasize that it's good to test training speed on the 6 core single machine cluster vs 10 core multi-core cluster). There are instructions for that in the H2O user guide here. There is also an FAQ about H2O clusters here. If the docs are not clear enough, please let me know (I am noticing it's a bit sparse on examples/information).

Lastly, if you specifically want to use Amazon EC2 for clustering, there's more info on that here.

Upvotes: 0

Related Questions