How to disable SSL authentication when fetching Spark packages?

Question

I am using pyspark and when I initiate my Spark session I fetch some packages from a Nexus internal repository using the spark.jars.package. To fetch the packages I set:

• spark.jars.ivySettings to point to a XML file with the locations in the Nexus internal repo to fetch the packages

• spark.driver.extraJavaOptions set to -Djavax.net.ssl.trustStore= to point to JKS file containing the certificates needed for SSL verification in the Nexus internal repo

This setup was working well for me until recently, when I started getting SSL authentication errors of the type:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)

I am checking with platform team what is causing these errors, but meanwhile, I would like to simply disable SSL verification for the purpose of fetching packages using spark.jars.packages, since there is no security risk as the system is properly ringfenced and we already disable SSL authentication for other purposes such as installing pip packages or downloading files from Nexus using wget/curl.

Is there some way to do so via the Spark configuration?

Answer 1

You can pass in SPARK options:

-Dcom.sun.net.ssl.checkRevocation=false