DIY HPC cluster to run Jupyter/Python notebooks

Question

I recently migrated my Python / Jupyter work from a macbook to a refurbrished Gen 8 HP rackmounted server (192GB DDR3 2 x 8C Xeon E5-2600), which I got off amazon for $400. The extra CPU cores have dramatically improved the speed of fitting my models particularly for decision tree ensembles that I tend to use a lot. I am now thinking of buying additional servers from that era (early-mid 2010s) (either dual or quad-socket intel xeon E5, E7 v1/v2) and wiring them up as a small HPC cluster in my apartment. Here's what I need help deciding:

1. Is this a bad idea? Am I better off buying a GPU (like a gtx 1080). The reason I am reluctant to go the GPU route is that I tend to rely on sklearn a lot (that's pretty much the only thing I know and use). And from what I understand model training on gpus is not currently a part of the sklearn ecosystem. All my code is written in numpy/pandas/sklearn. So, there will be a steep learning curve and backward compatibility issues. Am I wrong about this?

2. Assuming (1) is true and CPUs are indeed better for me in the short term. How do I build the cluster and run Jupyter notebooks on it. Is it as simple as buying an additional server. Designating one of the servers as the head node. Connecting the servers through ethernet. Installing Centos / Rocks on both machines. And starting the Jupyter server with IPython Parallel (?).

3. Assuming (2) is true, or at least partly true. What other hardware / software do I need to get? Do I need an ethernet switch? Or if I am connecting only two machines, there's no need for it? Or do I need a minimum of three machines to utilize the extra CPU cores and thus need a switch? Do I need to install Centos / Rocks? Or are there better, more modern alternatives for the software layer. For context, right now I use openSUSE on the HP server, and I am pretty much a rookie when it comes to operating systems and networking.

4. How homogeneous should my hardware be? Can I mix and match different frequency CPUs and memory across the machines? For example, having 1600 MHz DDR3 memory in one machine, 1333 MHz DDR3 in another? Or using 2.9 GHz E5-2600v1 and 2.6 GHz E5-2600v2 CPUs?

5. Should I be worried about power? I.e. can I safely plug three rackmounted servers in the same power strip in my apartment? There's one outlet that I know if I plug my hairdryer in, the lights go out. So I should probably avoid that one :) Seriously, how do I run 2-3 multi-CPU machines under load and avoid tripping the circuit breaker?

Thank you.

Answer 1

I am writing an answer to this post not necessarily because I think it's right or wrong to do it one way or another, but because there's only one answer to this question, and in reality Amazon and Microsoft are not your only options. You can do it yourself, I did. I wanted to share that experience.

I have a HPC in my home. I also come from a background at a computer repair shop, so there is not a problem which I cannot diagnose and solve somehow. I have years of experience and industry certifications in the area and that helps me to understand what's necessary, and build this out on my own.

I found the barrier to entry is very high, for multiple reasons, which are:

1). Patience

2). Time

3). But not really money.

You can build a really nice cluster for cheap. 10 Lenovo Thinkcentres cost $50 each. Server hardware is more expensive, but is better and more reliable.

4). This job is usually done by multiple people, because there is so many different domains of knowledge you must know. Sheer burden.

5). Sheer persistence.

If you want real cluster experience, then there is no better way to learn than to build one yourself.

I think all the questions you asked are relevant.

1). No. Build the hardware to the software. Code is expensive. Hardware is cheap. You should build the solution that allows you to innovate the best and most cost effective way for you. It may be the case you need more power for longer than you can afford to rent. It may be that you need the experience.

2). You can start with Jupyter, but what you will eventually be able to learn is Kubernetes, Spark, Etc and other technologies that are actually enterprise worthy, but while you're prototyping, use what's comfortable. You expect you to learn these technologies without exposure. It's an iterative process.

3). You will need switches. Different kinds of them, and my recommendation is you try to do it 10 gig if you can. I recommend you use link aggregation and stacked switches.

4). Yes you can mix, but it makes it harder to debug.

5). The difference between the power consumption of newer model machines and older model machines is marginal. What is not marginal is that a $30,000 machine five years ago is worth garbage, and you can buy a lot of them for cheap. If you get five and one is bad, no big deal, you have a parts machine and buy another. Also, that one machine can run hundreds of micro instances which each cost $25 a month. A starter kubernetes cluster on AWS is $30 a day, minimum. It adds up, fast.

Some points:

Computer hardware doesn't just magically fail after five years. Theres a reason it fails. That reason is it's environment, and because of wear and tear.

You can replace all the wear components, the key is to buy enough of them, and to build your system in a fault tolerant way. These servers usually come bare bones with no disks anyway, so your failures are removed.

Drive failures are the most common reason for server crashes. Second is power failures. Drive failures you can guard against by replacing them when the likelihood of failure is relatively high, its a calculated risk but if youre running a raidz3 or raidz2 and have boxes of disks you really don't care. Power failures require a generator to 100% guard against, but most of the time a battery will mitigate most of them.

It pays off in the long run, with AWS -- to have any decent power -- you're spending a lot of money. The equivalent server you can get for $400 once, dollars per hour; so dollar for dollar the math becomes very easy. The power bills are no more than an extra $200 a month for $100,000 a year worth of computing power.

You can lookup and swap any component on that system. The chief advantage of using server hardware is that it is meant to be serviced and is meant to be reliable.

You need some other hardware too. You'll quickly run into things like not being able to move enough packets between the servers. You can get other kinds of refurbished equipment that is cheap also, and you will need:

• Rack
• Multiple PDUS
• (New) One battery each 1500W
• Dedicated 20A Circuits.
• Rack hardware.
• Cable tools (crimpers, ethernet cable, etc)
• Servers
• USB
• Laptop
• Lots of hard drives.
• Namecheap / API compatible hosting account
• SFP's

Some general tips:

a). Pay really close attention to IP addressing, and use patterns you can script.

b). Always use an odd number of nodes.

c). When you boot up a cluster, do it one node at a time.

d). It's not meant to be shut down, but if you must, stop all vms and, do it one node at a time.

e). One of the first things you will do is setup cluster networking, and come up with a hostname, you will use a combination of routing based on DNS for kubernetes, and routing based on IP. Be consistent, use numbers in the names you can script.

f). Heavily leverage Dancer's Shell and

for i in $(seq end -1 start); do ssh -t root@server$i "command";

g). Expect to rebuild it multiple times

h). Spend the extra money to upgrade that puny 8C chip to a 12C one. Be careful when inserting the processor, don't smear the paste, just a pea sized amount.

i). You will come across 520 sector disks. You can fix those easily with sgdisk but you need the right HBA to do it. Any used LSI will work.

My Stack:

• Proxmox PVE
• Turnkey Linux VM
• Windows DC x 2
• Pihole x 2
• Ansible - Key
• Nginx (HA)
• Jenkins
• Kubernetes (39 t2 micro instances)
• MetalLB
• Greylog
• Hashicorp Vault
• Postgres
• Redis
• Mysql
• Spark
• Rabbit
• Kafka
• Jupyter
• Git
• Jenkins
• Nexus

I setup my network like this:

172.16.0.0/16 for cluster OOB subnet for kube

172.16.x.y where x corresponds to pve # and y corresponds to node on the pve. 172.16.255.0/24 for all floating nodes 172.16.254.0/24 for core infra. 172.16.253.0/24 for kubernetes load balancer

Plenty of room to grow.

For the hardware I have:

1 NAS 2 Storage Servers 2 Compute Servers 4 GPU servers 11 Desktops. 1 Aggregation Switch 1 24p 2 8p. 1 Omada Integrated Router

My next move is to setup three clusters, and link them over geographic locations by site to site.

I have 5 more compute server nodes to add. During this process I have learned a lot and really got to understand what is happening when I provision cloud resources. I highly recommend any enthusiast check this out.

Answer 2

1. Nvidia's rapids.ai implements a fair bit of sklearn on gpus. Whether that is the part you use, only you can say.

2. Using Jupiter notebooks for production is known to be a mistake.

3. You don't need a switch unless latency is a serious issue, it rarely is.

4. Completely irrelevant.

5. For old hardware of the sort you are considering, you will be having VERY high power bills. But worse, since you will have many not-so-new machines, the probability of some component failing at any given time is high, so unless you seek a future in computer maintenance, this is not a great idea. A better idea is: develop your idea on your macbook/existing cluster, then rent an AWS spot instance (or two or three) for a couple of days. Cheaper, no muss, no fuss. everything just works.