Can I do a setup like Azure HdInsight on Azure virtual machine?

Question

I would like to have a setup like Azure HDInsight on Azure VM and where I have to install and configure Hadoop to maintain the entire Hadoop and spark ecosystem, and most important thing is, I can only pay for a virtual machine, not for the software I have installed and I should achieve the same performance an HDInsight is that possible.

Answer 1

Can I do a setup like Azure HDInsight on Azure Virtual Machine?

Yes, you can setup like Azure HDInsight on Azure Virtual Machine.

Note: Azure HDInsight is built on top of Hortonworks Data platform.

You have four different options to deploy Hadoop.

1. Azure HDInsights (Built on top of HDP)

Azure HDInsight is a managed, full-spectrum, open-source analytics service for enterprises. HDInsight is a cloud service that makes it easy, fast, and cost-effective to process massive amounts of data. HDInsight also supports a broad range of scenarios, like extract, transform, and load (ETL); data warehousing; machine learning; and IoT.

2. HDP as a Service in Azure Marketplace

The Azure cloud infrastructure has become a common place for users to deploy virtual machines on the cloud due to its flexibility, ease of deployment, and cost benefits. Microsoft has expanded Azure to include a marketplace with thousands of certified, open source, and community software applications and developer services, pre-configured for Microsoft Azure. This tutorial covers deploying the Hortonworks Sandbox offering via the Azure Marketplace.

For a step-by-step guide on how to deploy the Hortonworks Sandbox on Azure, visit: Deploying Hortonworks Sandbox on Microsoft Azure.

3. Deploying HDP on bare metal

The sandbox provides a local development environment to learn about Hadoop, Hadoop Distributed File System (HDFS), and job submission.

For a step-by-step guide on Get started with an Apache Hadoop Sandbox, an emulator on a Virtual Machine.

4. Create a Linux VM and install Hadoop eco-system manually.

You can setup Hadoop eco-system on Azure Virtual Machine running linux. For a step-by-step guide on how to install Hadoop on Azure VM.

• Introduction
• Step-by-Step: Build the Infrastructure
• Step-by-Step: Install a Hadoop Distribution

Hope this helps.

Answer 2

As HDinsight is based on Hortonworks HDP, you might be able to get to a fairly similar system by installing something like the HDP sandbox on a VM: https://www.cloudera.com/downloads/hortonworks-sandbox.html

Generally: You can of course always just provision VM(s) and install an entire Hadoop distribution of your choice in there which gives you the full freedom to do and configure anything you like in there. If you get to the same performance as HDinsight depends then largely on your ability to properly configure and scale an Hadoop cluster.