CODEWITHSUNDEEP
scalaapache-sparkspark-graphx
Programmerr
Programmerr

Reputation: 1041

GraphX not working properly Spark / Scala

I am trying to create a GraphX object in apache Spark/Scala but it doesn't seem to be working for some reason. I have attached a file of the example input file, the actual program code is:

package SGraph

import org.apache.spark._

import org.apache.spark.SparkContext._

import org.apache.spark.sql._

import org.apache.log4j._

import org.apache.spark.rdd.RDD

import org.apache.spark.graphx._
`
    object GooglePlusGraph {

  /** Our main function where the action happens */
      def main(args: Array[String]) {

    // Set the log level to only print errors
    Logger.getLogger("org").setLevel(Level.ERROR)

    // Create a SparkContext using every core of the local machine
    val sc = new SparkContext("local[*]", "GooglePlusGraphX") 

    val lines = sc.textFile("../Example.txt")

    val ratings = lines.map(x => x.toString().split(":")(0))

    val verts  = ratings.map(line => (line.toLong,line)) 

    val edges = lines.flatMap(makeEdges)  

    val default = "Nobody"
    val graph = Graph(verts, edges, default).cache()

    graph.degrees.join(verts).take(10).foreach(println)
    }

    def makeEdges(line: String) : List[Edge[Int]] = {

    import scala.collection.mutable.ListBuffer

    var edges = new ListBuffer[Edge[Int]]()

    val fields = line.split(",").flatMap(a => a.split(":"))

    val origin = fields(0)

    for (x <- 1 to (fields.length - 1)) {
      // Our attribute field is unused, but in other graphs could
      // be used to deep track of physical distances etc.
      edges += Edge(origin.toLong, fields(x).toLong, 0)
    }

    return edges.toList
  }
}

The first error i get is the following:

16/12/19 01:28:33 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 3)
java.lang.NumberFormatException: For input string: "935750800736168978117"

thanks for any help !

Upvotes: 1

Views: 295

Answers (1)

Dongjoon Hyun
Dongjoon Hyun

Reputation: 64

It's the same issue with the following your question.

Cannot convert string to a long in scala

The given number has 21 digits beyond the maximum number of digits of Long (19 digits).

Upvotes: 1

Related Questions