CODEWITHSUNDEEP
apache-sparkdatasource
Wei
Wei

Reputation: 371

how to write spark custom data source based on FileFormat

i saw the spark avro datasource is implemented based on FileFormat interface. Is there any documentation about how to write spark custom datasource based on FileFormat? Up to now i can't find any(except the source code from spark avro).

Thank you!

Upvotes: 9

Views: 5611

Answers (1)

yobibytes
yobibytes

Reputation: 154

Here is an example of a simple file-based spark datasource: https://hackernoon.com/extending-our-spark-sql-query-engine-5f4a088de986

Here's a couple examples that implement the Data Sources API, as well: * https://github.com/databricks/spark-csv * https://github.com/databricks/spark-avro

Upvotes: 6

Related Questions