CODEWITHSUNDEEP
apache-sparkspark-streamingdatastax
Ryan Stack
Ryan Stack

Reputation: 1331

Hide a spark property from displaying in the spark web UI without implementing a security filter

The application web UI at http://:4040 lists Spark properties in the “Environment” tab. All values explicitly specified through spark-defaults.conf, SparkConf, or the command line will appear. However, for security reasons, I do not want my Cassandra password to display in the web UI. Is there some sort of switch to ensure that certain spark properties are not displayed??

Please note, I see some solutions that suggest implementing a security filter and using spark.ui.filters setting to refer to the class. I am hoping to avoid this complexity.

Upvotes: 4

Views: 1578

Answers (2)

Pavel Orekhov
Pavel Orekhov

Reputation: 2177

You need to use spark.redaction.regex in spark conf. You can look here for the docs: https://spark.apache.org/docs/3.5.2/configuration.html

Regex to decide which Spark configuration properties and environment variables in driver and executor environments contain sensitive information. When this regex matches a property key or value, the value is redacted from the environment UI and various logs like YARN and event logs.

Upvotes: 0

VladoDemcak
VladoDemcak

Reputation: 5259

I think there is no common solution how to hide your custom property from spark WebUI for previous releases.

I assume you are using spark 2.0 or below (i have not seen feature described below in 2.0) because 2.0.1 supports passwords preprocessing to "*****".

Check issue SPARK-16796 Visible passwords on Spark environment page

When we take a look into apache spark source code and do some investigation we can see some processing how to "hide" property in spark web ui.

SparkUI by default the Environment page is attached within initialization attachTab(new EnvironmentTab(this)) [line 71]

EnvironmentPage renders properties to EnvironmentPage as tab in web gui as:

def render(request: HttpServletRequest): Seq[Node] = {
    val runtimeInformationTable = UIUtils.listingTable(
      propertyHeader, jvmRow, listener.jvmInformation, fixedWidth = true)
    val sparkPropertiesTable = UIUtils.listingTable(
      propertyHeader, propertyRow, listener.sparkProperties.map(removePass), fixedWidth = true)
    val systemPropertiesTable = UIUtils.listingTable(
      propertyHeader, propertyRow, listener.systemProperties, fixedWidth = true)
    val classpathEntriesTable = UIUtils.listingTable(
      classPathHeaders, classPathRow, listener.classpathEntries, fixedWidth = true)
    val content =
      <span>
        <h4>Runtime Information</h4> {runtimeInformationTable}
        <h4>Spark Properties</h4> {sparkPropertiesTable}
        <h4>System Properties</h4> {systemPropertiesTable}
        <h4>Classpath Entries</h4> {classpathEntriesTable}
      </span>

    UIUtils.headerSparkPage("Environment", content, parent)
  }

all properties are rendered without some kind of hiding preprocessing except sparkProperties - with functionality provided in removePass.

private def removePass(kv: (String, String)): (String, String) = {
    if (kv._1.toLowerCase.contains("password")) (kv._1, "******") else kv
}

as we can see every key that contains "password" (BTW: in the master branch they also filtering keys with keyword "secret" check if u are interested in)

I cannot tested now but u can try to update spark. so eg. SparkSubmitArguments.scala in mergeDefaultSparkProperties() will consider spark.cassandra.auth.password as spark and populate as sparkProperties (with removePass preprocessing).

And at the end of the day in EnvironmentTab in web gui this property should be visible as ****.

Upvotes: 2

Related Questions