CODEWITHSUNDEEP
scalaapache-sparkdataframedataset
saravanan saminathan
saravanan saminathan

Reputation: 702

Difference between File write using spark and scala and the advantages?

DF().write
  .format("com.databricks.spark.csv")
  .save("filepath/selectedDataset.csv")

vs

scala.tools.nsc.io.File("/Users/saravana-6868/Desktop/hello.txt").writeAll("String"))

In the above code, I used to write a file using both dataframes and scala. What is the difference in the above code?

Upvotes: 0

Views: 188

Answers (1)

Chandan Ray
Chandan Ray

Reputation: 2091

The first piece of code is specific to SPARK API of writing the dataframe to a file in csv format. You can write to hdfs or local file system using this. even you can repartition and parallellize your write. The second piece of code is SCALA API which can only write in the local file system. You cannot parallelize it. The first code levearage the whole cluster to do its work but not the second piece of code.

Upvotes: 1

Related Questions