CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sqlapache-spark-1.6spark-submit
sudheeshix
sudheeshix

Reputation: 1591

Which jar has org.apache.spark.sql.types?

I am on Spark 1.x, and attempting to read csv files. If I need to specify some data types, as per the documentation, I need to import the types defined in the package org.apache.spark.sql.types.

import org.apache.spark.sql.types.{StructType,StructField,StringType};

This works fine when I use this interactively in spark-shell, but as I want to run this thru spark-submit, I wrote some Scala code to do this. But, when I attempt to compile my Scala code, it gives me an error saying it could NOT find org.apache.spark.sql.types. I looked up the jar contents of spark-sql, but couldn't find these types defined in there.

So, which jar has org.apache.spark.sql.types?

Upvotes: 0

Views: 3207

Answers (1)

sudheeshix
sudheeshix

Reputation: 1591

I looked at the source code for spark-sql at GitHub to realize that these types can be found in the spark-catalyst jar. That didn't seem intuitive.

Also, since StructType has this code

org.json4s.JsonDSL._

we end up with another dependent jar - json4s-core.

Upvotes: 2

Related Questions