How can I resolve dependency com.typesafe.play#play-json;2.4.0

Question

I am trying to run my jar file in the linux terminal of my local machine using spark-submit command

> spark-submit \
> --master local \
> --class RTP \
> --packages com.typesafe:config:1.3.2,org.apache.kafka:kafka-clients:2.0.0,
    commons-codec:commons-codec:1.9,org.scalaj:scalaj-http_2.11:2.3.0,
    com.typesafe.play:play-json:2.4.0 \
> target/scala-2.11/structured_streaming_2.11-0.1.jar

However the program fails and I get the following output

Ivy Default Cache set to: /home/user/.ivy2/cache
The jars for the packages stored in: /home/user/.ivy2/jars
:: loading settings :: url = jar:file:/home/user/spark-2.3.3-bin-hadoop2.7/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.typesafe#config added as a dependency
org.apache.kafka#kafka-clients added as a dependency
commons-codec#commons-codec added as a dependency
org.scalaj#scalaj-http_2.11 added as a dependency
com.typesafe.play#play-json added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-975a9ed1-7824-48a2-81df-93814b2aa810;1.0
    confs: [default]
    found com.typesafe#config;1.3.2 in spark-list
    found org.apache.kafka#kafka-clients;2.0.0 in spark-list
    found org.lz4#lz4-java;1.4.1 in spark-list
    found org.xerial.snappy#snappy-java;1.1.7.1 in spark-list
    found org.slf4j#slf4j-api;1.7.25 in spark-list
    found commons-codec#commons-codec;1.9 in spark-list
    found org.scalaj#scalaj-http_2.11;2.3.0 in spark-list
:: resolution report :: resolve 905ms :: artifacts dl 6ms
    :: modules in use:
    com.typesafe#config;1.3.2 from spark-list in [default]
    commons-codec#commons-codec;1.9 from spark-list in [default]
    org.apache.kafka#kafka-clients;2.0.0 from spark-list in [default]
    org.lz4#lz4-java;1.4.1 from spark-list in [default]
    org.scalaj#scalaj-http_2.11;2.3.0 from spark-list in [default]
    org.slf4j#slf4j-api;1.7.25 from spark-list in [default]
    org.xerial.snappy#snappy-java;1.1.7.1 from spark-list in [default]
    ---------------------------------------------------------------------
    |                  |            modules            ||   artifacts   |
    |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
    ---------------------------------------------------------------------
    |      default     |   8   |   0   |   0   |   0   ||   7   |   0   |
    ---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
        module not found: com.typesafe.play#play-json;2.4.0

    ==== local-m2-cache: tried

      file:/home/sushil/.m2/repository/com/typesafe/play/play-json/2.4.0/play-json-2.4.0.pom

      -- artifact com.typesafe.play#play-json;2.4.0!play-json.jar:

      file:/home/sushil/.m2/repository/com/typesafe/play/play-json/2.4.0/play-json-2.4.0.jar

    ==== local-ivy-cache: tried

      /home/sushil/.ivy2/local/com.typesafe.play/play-json/2.4.0/ivys/ivy.xml

      -- artifact com.typesafe.play#play-json;2.4.0!play-json.jar:

      /home/sushil/.ivy2/local/com.typesafe.play/play-json/2.4.0/jars/play-json.jar

    ==== central: tried

      https://repo1.maven.org/maven2/com/typesafe/play/play-json/2.4.0/play-json-2.4.0.pom

      -- artifact com.typesafe.play#play-json;2.4.0!play-json.jar:

      https://repo1.maven.org/maven2/com/typesafe/play/play-json/2.4.0/play-json-2.4.0.jar

    ==== spark-packages: tried

      http://dl.bintray.com/spark-packages/maven/com/typesafe/play/play-json/2.4.0/play-json-2.4.0.pom

      -- artifact com.typesafe.play#play-json;2.4.0!play-json.jar:

      http://dl.bintray.com/spark-packages/maven/com/typesafe/play/play-json/2.4.0/play-json-2.4.0.jar

        ::::::::::::::::::::::::::::::::::::::::::::::

        ::          UNRESOLVED DEPENDENCIES         ::

        ::::::::::::::::::::::::::::::::::::::::::::::

        :: com.typesafe.play#play-json;2.4.0: not found

        ::::::::::::::::::::::::::::::::::::::::::::::



:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: com.typesafe.play#play-json;2.4.0: not found]
    at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1303)
    at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:53)
    at org.apache.spark.deploy.SparkSubmit$.doPrepareSubmitEnvironment(SparkSubmit.scala:364)
    at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:250)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:171)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

When I run the spark-submit version command on my local machine I get the following

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.3.3
      /_/
Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_252

This program runs perfectly fine on Intellij

The details of my build.sbt file are as follows

name := "Structured_Streaming"

version := "0.1"

scalaVersion := "2.11.12"

libraryDependencies += "com.typesafe" % "config" % "1.3.2"
libraryDependencies += "org.apache.kafka" % "kafka-clients" % "2.0.0"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.4.0"
libraryDependencies += "org.apache.spark" % "spark-sql-kafka-0-10_2.11" % "2.4.0"
libraryDependencies += "commons-codec" % "commons-codec" % "1.9"

libraryDependencies += "org.scalaj" % "scalaj-http_2.11" % "2.3.0"


libraryDependencies += "com.typesafe.play" %% "play-json" % "2.4.0"

Answer 1

In --packages you have com.typesafe.play:play-json:2.4.0 instead of com.typesafe.play:play-json_2.11:2.4.0 so you are fetching content from

https://repo1.maven.org/maven2/com/typesafe/play/play-json/2.4.0/

instead of

https://repo1.maven.org/maven2/com/typesafe/play/play-json_2.11/2.4.0/