Is it possible to load a Scala/Spark PipelineModel by mlflow?

Question

I try to use mlflow to load a serialized Scala/Spark PipelineModel.

From mlflow document it seems it only supports PySpark. Is this the case?

Is it possible to load a serialized Scala/Spark PipelineModel in mlflow?

Answer 1

MLflow has a Java client which can be called from Scala. See: https://mlflow.org/docs/latest/java_api/index.html

The Java client has less features than the Python client in that it does not have the concept of MLflow flavors. So it doesn't have a flavor-specific log_model/logModel - you have to use logArtifact.

Example:

• https://github.com/amesar/mlflow-examples/tree/master/scala/sparkml https://github.com/amesar/mlflow-examples/tree/master/scala/sparkml/src/main/scala/org/andre/mlflow/examples/wine/sparkml