Errors Creating Simple Azure HDInsight Spark Cluster with Pulumi

Question

I am attempting to use the Pulumi Javascript SDK to create a HDInsight Spark Cluster on Azure. I have followed the tutorials Pulumi provides on creating a "hello world" GCP Kubernetes cluster and gone through the JavaScript examples in the Pulumi Examples repo on github but have had no luck in successfully creating the cluster.

I have attempted changing my export statements multiple times based on the examples in the Pulumi repo but I always get an "unhandled exception" error followed by a stacktrace when I run pulumi up inside my pulumi project root directory. I have tried using the following export statements so far.

    //attempt 1
    export const sparkClusterName = sparkCluster.name

    //attempt 2
    export const sparkClusterOutput = sparkCluster.output

    //attempt 3
    export const sparkEndpoint = sparkCluster.httpsEndpoint

I'm using Visual Studio Code as my IDE Running all code on a MacBook Pro with Mojave using Pulumi version 0.17.11 using node version 12.1.0 the azure cli and pulumi cli tools are both installed

My index.js program is as follows:

"use strict";
const pulumi = require("@pulumi/pulumi");
const azure = require("@pulumi/azure");

// Retrieve an Azure Resource Group
const resourceGroup = pulumi.output(azure.core.getResourceGroup({
    name: "MyResourceGroup",
}));
//Create Spark Cluster
const sparkCluster = new azure.hdinsight.SparkCluster("testSparkCluster", {
    clusterVersion: "3.6",
    componentVersion: {
        spark: "2.3",
    },
    gateway: {
        enable: true,
        password: "laDK#21",
        username: "USERname",
    },
    location: resourceGroup.apply(resourceGroup => resourceGroup.location),
    name: "example-hdisparkcluster",
    resourceGroupName: resourceGroup.apply(resourceGroup => resourceGroup.name),
    roles: {
        headNode: {
            password: "AAAlllddck11122$$3",
            username: "USerNameladkfj",
            vmSize: "Standard_A3",
        },
        workerNode: {
            password: "asdlfaDDF143k#@#",
            targetInstanceCount: 3,
            username: "USernaemls",
            vmSize: "Standard_A3",
        },
        zookeeperNode: {
            password: "ASDKLlkjjj##@@323",
            username: "USERname2323",
        },
    },
    storageAccounts: [{
        isDefault: true,
    }],
    tier: "Standard",
});

// Export the spark cluster
export const sparkClusterName = sparkCluster.name;
//export const sparkName = sparkCluster.output
//export const sparkEndpoint = sparkCluster.httpsEndpoint

The expected result should be a success from pulumi CLI showing the new stack and Spark Cluster being created (and then being able to view the cluster on the Azure Portal). Instead the following error and stack trace is thrown:

Previewing update (dev):

     Type                 Name                         Plan       Info
 +   pulumi:pulumi:Stack  HDInsight_Spark_Cluster-dev  create     1 error

Diagnostics:
  pulumi:pulumi:Stack (HDInsight_Spark_Cluster-dev):
    error: Running program '/Users/workspace/Pulumi Workspace/Pulumi HDInsight Testing' failed with an unhandled exception:
    /Users/workspace/Pulumi Workspace/Pulumi HDInsight Testing/index.js:50
    export const sparkClusterName = sparkCluster.name;
    ^^^^^^

    SyntaxError: Unexpected token export
        at Module._compile (internal/modules/cjs/loader.js:703:23)
        at Object.Module._extensions..js (internal/modules/cjs/loader.js:770:10)
        at Module.load (internal/modules/cjs/loader.js:628:32)
        at Function.Module._load (internal/modules/cjs/loader.js:555:12)
        at Module.require (internal/modules/cjs/loader.js:666:19)
        at require (internal/modules/cjs/helpers.js:16:16)
        at /Users/workspace/Pulumi Workspace/Pulumi HDInsight Testing/node_modules/@pulumi/pulumi/cmd/run/run.js:195:20
        at Stack.<anonymous> (/Users/workspace/Pulumi Workspace/Pulumi HDInsight Testing/node_modules/@pulumi/pulumi/runtime/stack.js:76:27)
        at Generator.next (<anonymous>)
        at fulfilled (/Users/workspace/Pulumi Workspace/Pulumi HDInsight Testing/node_modules/@pulumi/pulumi/runtime/stack.js:17:58)

I'm very new to using Azure and have only a basic understanding of Javascript as most of my experience is with Java and C/C++. My purpose for this program is just helping me to use and understand working with HDInsight by using Pulumi.

Answer 1

You are hitting a syntax issue because exports in JavaScript are different from TypeScript. Your copied export would work in a TS program, but in JS you should write:

exports.sparkClusterName = sparkCluster.name;

If you are new to both JavaScript and TypeScript, I suggest you switching to TypeScript and using an editor like Visual Studio Code to highlight errors at edit time. There are still a number of those in your code: enable instead of enabled, vmSize missing in zookeeperNode, storageAccountKey and storageContainerId missing in storageAccounts.

On top of that, your usernames and password are violating some policies, but you would only discover that while running pulumi up, not at edit time.

Here is my take:

const sparkCluster = new azure.hdinsight.SparkCluster("testSparkCluster", {
    clusterVersion: "3.6",
    componentVersion: {
        spark: "2.3",
    },
    gateway: {
        enabled: true,
        password: "@P6g4KMvlhjM",
        username: "username",
    },
    name: "example-hdisparkcluster",
    resourceGroupName: resourceGroup.apply(rg => rg.name),
    roles: {
        headNode: {
            password: "0@Gkv81xt!lR",
            username: "usernamehn",
            vmSize: "Standard_A3",
        },
        workerNode: {
            password: "Wx63!ZMnePHK",
            targetInstanceCount: 3,
            username: "usernamewn",
            vmSize: "Standard_A3",
        },
        zookeeperNode: {
            password: "&43MhoXfZ5ar",
            username: "usernamezn",
            vmSize: "Standard_A3",
        },
    },
    storageAccounts: [{
        isDefault: true,
        storageAccountKey: storageAccount.primaryAccessKey,
        storageContainerId: storageContainer.id,
    }],
    tier: "Standard",
});

UPDATE: I created a full working example.