CODEWITHSUNDEEP
apache-spark
user3376961
user3376961

Reputation: 877

Spark: Importing Data

I currently have a spark app that reads a couple of files and forms a data frame out of them and implements some logic on the data frames.

I can see the number and size of these files growing by a lot in the future and wanted to understand what goes on behind the scenes to be able to keep up with this growth.

Firstly, I just wanted to double check that since all machines on the cluster can access the files (which is a requirement by spark), the task of reading in data from these files is distributed and no one machine is burdened by it? I was looking at the Spark UI for this app but since it only shows what actions were performed by which machines and since "sc.textFile(filePath)" is not an action I couldn't be sure what machines are performing this read.

Secondly, what advantages/disadvantages would I face if I were to read this data from a database like Cassandra instead of just reading in files?

Thirdly, in my app I have some code where I perform a collect (val treeArr = treeDF.collect()) on the dataframe to get an array and then I have some logic implemented on those arrays. But since these are not RDDs, how does Spark distribute this work? Or does it distribute them at all? In other words, should I be doing maximum amount of my work transforming and performing actions on RDDs than converting them into arrays or some other data structure and then implementing the logic like I would in any programming language?

I am only about two weeks into Spark so I apologize if these are stupid questions!

Upvotes: 0

Views: 165

Answers (1)

Daniel Darabos
Daniel Darabos

Reputation: 27470

  1. Yes, sc.textFile is distributed. It even has an optional minPartitions argument.

  2. This question is too broad. But the short answer is that you should benchmark it for yourself.

  3. collect fetches all the data to the master. After that it's just a plain array. Indeed the idea is that you should not use collect if you want to perform distributed computations.

Upvotes: 1

Related Questions