CODEWITHSUNDEEP
apache-sparkspark-streaming
Suzy Tros
Suzy Tros

Reputation: 373

from single machine to parallel processing?

i am new to Spark Streaming and Big Data in general. I am trying to understand the structure of a project in spark. I want to create a main class lets say "driver" with M machines, each machine keeps an array of counters and its values. In single machine and not in Spark, I would create a class for the machines and a class for the counters and i would do the computations that i want. But i am wondering if that is happening in Spark too. Would the same project but in Spark, have the structure I am quoting bellow?

class Driver {
    var num : Int = 100
    var machines: Array[Machine] = new Array[Machine](num)
    //split incoming dstream and fill machines' queues
}
class Machine {
 var counters = new Queue[(Int,Int)]() // counter with id 1 and value 25
 def fillCounters: Unit = { ... } //function to fill the queue counters
}

Upvotes: 0

Views: 95

Answers (1)

gemelen
gemelen

Reputation: 617

In general, you could imagine Spark application as a driver part of application which runs all coordination tasks, constructs graph (you will find mentions of direct acyclic graph or DAG in theoretical parts of tutorials on Spark and distributed computations) of operations to take place over your data, and executor part which results in many copies of the code, sent to each node of the cluster to run over the data.

Main idea is that driver extracts part of your application's code that needs to be run locally with data on nodes, serializes it, sends over network to each executor, launches, manages and collects results.

Spark framework hides this details for simplicity of usage, so applications being developed and looks like single-threaded application.

Developer could separate contexts that run on driver and executors, but this is not very common for tutorials (again, for simplicity).

So, to the answer for the actual question above: you do not need to design your application in a way you demonstrated above, unless your certainly want to. Just follow official Spark tutorial to get viable solution and split it afterwards with contexts of execution.

There is good post, summarizing a lot of Spark turorials, videos and talks - you could find it here at SO.

Upvotes: 2

Related Questions