CODEWITHSUNDEEP
apache-sparkmachine-learningpysparklogistic-regressionsignificance
CARTman
CARTman

Reputation: 747

Can I extract significane values for Logistic Regression coefficients in pyspark

Is there a way to get the significance level of each coefficient we receive after we fit a logistic regression model on training data?

I was trying to find out a way and could not figure out myself.

I think I may get the significance level of each feature if I run chi sq test but first of all not sure if I can run the test on all features together and secondly I have numeric data value so if it will give me right result or not that remains a question as well.

Right now I am running the modeling part using statsmodel and scikit learn but certainly, want to know, how can I get these results from PySpark ML or MLLib itself

If anyone can shed some light, it will be helpful

Upvotes: 6

Views: 1624

Answers (1)

Rachid Ait Abdesselam
Rachid Ait Abdesselam

Reputation: 385

I use only mllib, I think that when you train a model you can use toPMML method to export your model un PMML format (xml file), then you can parse the xml file to get features weights, here an example

https://spark.apache.org/docs/2.0.2/mllib-pmml-model-export.html

Hope that will help

Upvotes: 4

Related Questions