CODEWITHSUNDEEP
apache-sparkpysparkcassandraamazon-emrspark-cassandra-connector
maxwellray
maxwellray

Reputation: 121

spark-cassandra-connector on EMR serverless (PySpark)

I'm working on getting an application running on EMR Serverless and am having trouble pulling in the spark-cassandra-connector. I have no issue pulling it on my local, but all of my attempts at using the library on EMR Serverless have failed.

When I include the library using --jars s3://XXX/XXXX/spark-cassandra-connector-driver_2.12-3.2.0.jar, I error out on the following line

d = spark \
    .read \
    .format("org.apache.spark.sql.cassandra") \
    .options(table="YYYY", keyspace="YYY") \
    .load()

with the error

py4j.protocol.Py4JJavaError: An error occurred while calling o121.load.
: java.lang.ClassNotFoundException: 
Failed to find data source: org.apache.spark.sql.cassandra. Please find packages at
http://spark.apache.org/third-party-projects.html

When I try to add the package using --packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0, the application times out with

com.datastax.spark#spark-cassandra-connector_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-5ee06249-c545-4b92-804f-ecedd322158a;1.0
    confs: [default]
:: resolution report :: resolve 524554ms :: artifacts dl 0ms
    :: modules in use:
    ---------------------------------------------------------------------
    |                  |            modules            ||   artifacts   |
    |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
    ---------------------------------------------------------------------
    |      default     |   1   |   0   |   0   |   0   ||   0   |   0   |
    ---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
        module not found: com.datastax.spark#spark-cassandra-connector_2.12;3.2.0

    ==== local-m2-cache: tried

      file:/home/hadoop/.m2/repository/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.pom

      -- artifact com.datastax.spark#spark-cassandra-connector_2.12;3.2.0!spark-cassandra-connector_2.12.jar:

      file:/home/hadoop/.m2/repository/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.jar

    ==== local-ivy-cache: tried

      /home/hadoop/.ivy2/local/com.datastax.spark/spark-cassandra-connector_2.12/3.2.0/ivys/ivy.xml

      -- artifact com.datastax.spark#spark-cassandra-connector_2.12;3.2.0!spark-cassandra-connector_2.12.jar:

      /home/hadoop/.ivy2/local/com.datastax.spark/spark-cassandra-connector_2.12/3.2.0/jars/spark-cassandra-connector_2.12.jar

    ==== central: tried

      https://repo1.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.pom

      -- artifact com.datastax.spark#spark-cassandra-connector_2.12;3.2.0!spark-cassandra-connector_2.12.jar:

      https://repo1.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.jar

    ==== spark-packages: tried

      https://repos.spark-packages.org/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.pom

      -- artifact com.datastax.spark#spark-cassandra-connector_2.12;3.2.0!spark-cassandra-connector_2.12.jar:

      https://repos.spark-packages.org/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.jar

        ::::::::::::::::::::::::::::::::::::::::::::::

        ::          UNRESOLVED DEPENDENCIES         ::

        ::::::::::::::::::::::::::::::::::::::::::::::

        :: com.datastax.spark#spark-cassandra-connector_2.12;3.2.0: not found

        ::::::::::::::::::::::::::::::::::::::::::::::


:::: ERRORS
    Server access error at url https://repo1.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.pom (java.net.ConnectException: Connection timed out (Connection timed out))

    Server access error at url https://repo1.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.jar (java.net.ConnectException: Connection timed out (Connection timed out))

    Server access error at url https://repos.spark-packages.org/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.pom (java.net.ConnectException: Connection timed out (Connection timed out))

    Server access error at url https://repos.spark-packages.org/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.jar (java.net.ConnectException: Connection timed out (Connection timed out))

I'm betting the --package issue comes from a firewall configuration problem, but I'm not seeing any way to open up access. As for the --jars issue, I'm not sure why pulling in the .jar isn't enough for Spark to recognize the org.apache.spark.sql.cassandra format.

Any help on either issue would be appreciated, thanks!

Upvotes: 2

Views: 664

Answers (1)

Alex Ott
Alex Ott

Reputation: 87164

Yes, the issue with --packages most probably due your egress settings that prevent accessing the Maven central.

To use --jars you need to specify all necessary jars, like, driver, connector, Java driver, etc. The easiest way to avoid this is to use so-called assembly build that is available on Maven Central too with com.datastax.spark:spark-cassandra-connector-assembly_2.12:3.2.0 coordinate. Just download referenced jar file.

Upvotes: 2

Related Questions