CODEWITHSUNDEEP
azure-databricks
yet_another_programmer
yet_another_programmer

Reputation: 327

Databricks spark monitoring on Azure for Spark 3.0.0 and Scala 2.12

I have a Azure Databricks cluster that runs a cluster with Databricks version 7.1 (Spark 3.0.0 and Scala 2.12). I'm trying to send the Azure Databricks application logs to the Azure monitor, so I'm following this documentation.

In order to do so, I'm trying to build the spark-listeners-loganalytics-1.0-SNAPSHOT.jar JAR file as described in the GitHub readme.

To compile it, I need to edit the POM to create a Maven profile that includes my current Spark and Scala versions, so I have the following profile:

    <profile>
        <id>scala-2.12_spark-3.0.0</id>
        <activation>
            <activeByDefault>true</activeByDefault>
        </activation>
        <properties>
            <commons.httpclient.version>4.5.12</commons.httpclient.version>
            <jetty.version>9.4.31.v20200723</jetty.version>
            <spark.version>3.0.0</spark.version>
            <scala.version>2.12.10</scala.version>
            <scala.compat.version>2.12</scala.compat.version>
        </properties>
    </profile>

I've only the spark and scala versions. However, with these versions, the code does not compile. It appears that this library only supports Spark 2.X and Scala 2.11.X. Is there any possibility of using the library with Spark 3.0.0 and Scala 2.12?

Any help would be greatly appreciated.

Upvotes: 1

Views: 657

Answers (1)

Dustin V
Dustin V

Reputation: 336

The library does not support Spark 3.0.0 and above yet. Issue 92 on the repo (https://github.com/mspnp/spark-monitoring/issues/92) shows the conversation and a comment that Microsoft will work on it this quarter but no timeline provided for when it will be available. One comment in the issue is from someone who created a branch and got it working, but some tests had to be removed so I am not using it on my project unless I put more work into understanding the changes.

Upvotes: 2

Related Questions