CODEWITHSUNDEEP
apache-sparklinear-algebraapache-spark-mllibjblas
tphi
tphi

Reputation: 21

Apache Spark - MLlib - Matrix multiplication

I'm trying to use MLlib for matrix multiplication problem.

I am aware that Spark MLLib uses native libraries, which need to be present on the nodes. (that it does not come with spark installation).

So I already installed libgfortran library on all nodes (I did the same as Apache Spark -- MlLib -- Collaborative filtering)

But then I still encounter this error when running on a cluster.

Lost task 0.3 in stage 2.0 (TID 11, ibm-power-6.dima.tu-berlin.de): java.lang.UnsatisfiedLinkError: org.jblas.NativeBlas.dgemm(CCIIID[DII[DIID[DII)V
    at org.jblas.NativeBlas.dgemm(Native Method)
    at org.jblas.SimpleBlas.gemm(SimpleBlas.java:247)
    .....

How can I solve this error?

Upvotes: 2

Views: 365

Answers (1)

sgvd
sgvd

Reputation: 3939

Spark hasn't used jblas for a while; as far as I can tell at the moment not since 1.4.0, which came out more than a year ago. The answer you linked to links to documentation of Spark 0.9.0, which is definitely ancient. So the simplest solution seems to be to use a more up to date version of Spark.

If that is not possible, or if you run into a situation where you have to use jblas again: it looks like you are using IBM PowerLinux hardware. Support for this platform was added to jblas in version 1.2.4, so you would have to make sure you are using at least that version.

Upvotes: 0

Related Questions