How to solve incompatible dependency conflict through shading for local run

Question

For my project, I want to use sttp to perform REST API calls, parse incoming json, load it into a dataframe and write object store. I added the following to my library dependency

"com.softwaremill.sttp.client" %% "core"   % "2.3.0"
"com.softwaremill.sttp.client" %% "json4s" % "2.3.0"

However, this causes a conflict between Spark and json4s libraries, which are incompatible. I am using Spark 3.1.1 which has a dependency on

"org.json4s" %% "json4s-jackson" % "3.7.0-M5"

which in turn has transitive dependencies on

"org.json4s" %% "json4s-core" % "3.7.0-M5"
"org.json4s" %% "json4s-ast" % "3.7.0-M5"

among others.

The result is sparksession fails to start complaining about

Exception in thread "main" java.lang.NoClassDefFoundError: org/json4s/JsonAST$JValue

I looked the following SO threads

Is it possible to use json4s 3.2.11 with Spark 1.3.0?

Shading over third party classes

and added the following to my build.sbt

ThisBuild / assemblyShadeRules := Seq(
    ShadeRule
      .rename("org.json4s.**" -> "json4sast.@0")
      .inLibrary(
        "org.json4s" % "json4s-ast"     % "4.0.3",
        "org.json4s" % "json4s-core"    % "4.0.3",
        "org.json4s" % "json4s-jackson" % "4.0.3",
        "org.json4s" % "json4s-scalap"  % "4.0.3"
      )

but it did not resolve the error. I have two questions:

Is shading applicable primarily when we run assembly? Or can it shade conflicting libraries so that I can run my code locally?