CODEWITHSUNDEEP
apache-sparkapache-spark-sqlapache-spark-mllib
ldl
ldl

Reputation: 156

How to use MLlib in spark SQL

Lately, I've been learning about spark sql, and I wanna know, is there any possible way to use mllib in spark sql, like :

select mllib_methodname(some column) from tablename;

here, the "mllib_methodname" method is a mllib method. Is there some example shows how to use mllib methods in spark sql?

Thanks in advance.

Upvotes: 1

Views: 1705

Answers (1)

Xiangrui Meng
Xiangrui Meng

Reputation: 46

The new pipeline API is based on DataFrames, which is backed by SQL. See

http://spark.apache.org/docs/latest/ml-guide.html

Or you can simply register the predict method from MLlib models as UDFs and use them in your SQL statement. See

http://spark.apache.org/docs/latest/sql-programming-guide.html#udf-registration-moved-to-sqlcontextudf-java--scala

Upvotes: 1

Related Questions