CODEWITHSUNDEEP
pysparkapache-spark-ml
Gun
Gun

Reputation: 179

How to calculate gini index in pyspark classification model using spark ML?

I am trying to calculate the gini index for a classification model done using GBTClassifier from the pyspark ml models. I cant seem to find a metrics which gives the roc_auc_score like the one in python sklearn.

Below is the code that I have used so far on databricks. I am currently using a dataset from the databricks 

%fs ls databricks-datasets/adult/adult.data

from pyspark.sql.functions import *
from pyspark.ml.classification import  RandomForestClassifier, GBTClassifier
from pyspark.ml.feature import StringIndexer, OneHotEncoderEstimator, VectorAssembler, VectorSlicer
from pyspark.ml import Pipeline
from pyspark.ml.evaluation import BinaryClassificationEvaluator,MulticlassClassificationEvaluator
from pyspark.mllib.evaluation import BinaryClassificationMetrics
from pyspark.ml.linalg import Vectors
from pyspark.ml.tuning import ParamGridBuilder, TrainValidationSplit

dataset = spark.table("adult")
# spliting the train and test data frames 
splits = dataset.randomSplit([0.7, 0.3])
train_df = splits[0]
test_df = splits[1]

def churn_predictions(train_df,
                     target_col, 
#                      algorithm, 
#                      model_parameters = conf['model_parameters']
                    ):
  """
  #Function attributes
  dataframe        - training df
  target           - target varibale in the model
  Algorithm        - Algorithm used 
  model_parameters - model parameters used to fine tune the model
  """

  # one hot encoding and assembling
  encoding_var = [i[0] for i in train_df.dtypes if (i[1]=='string') & (i[0]!=target_col)]
  num_var = [i[0] for i in train_df.dtypes if ((i[1]=='int') | (i[1]=='double')) & (i[0]!=target_col)]

  string_indexes = [StringIndexer(inputCol = c, outputCol = 'IDX_' + c, handleInvalid = 'keep') for c in encoding_var]
  onehot_indexes = [OneHotEncoderEstimator(inputCols = ['IDX_' + c], outputCols = ['OHE_' + c]) for c in encoding_var]
  label_indexes = StringIndexer(inputCol = target_col, outputCol = 'label', handleInvalid = 'keep')
  assembler = VectorAssembler(inputCols = num_var + ['OHE_' + c for c in encoding_var], outputCol = "features")
  gbt = GBTClassifier(featuresCol = 'features', labelCol = 'label',
                     maxDepth = 5, 
                     maxBins  = 45,
                     maxIter  = 20)

  pipe = Pipeline(stages = string_indexes + onehot_indexes + [assembler, label_indexes, gbt])
  model = pipe.fit(train_df)

  return model  

gbt_model = churn_predictions(train_df = train_df,
                     target_col = 'income')

#### prediction in test sample ####
gbt_predictions = gbt_model.transform(test_df)
# display(gbt_predictions)
gbt_evaluator = MulticlassClassificationEvaluator(
    labelCol="label", predictionCol="prediction", metricName="accuracy")

accuracy = gbt_evaluator.evaluate(gbt_predictions) * 100
print("Accuracy on test data = %g" % accuracy)

gini_train = 2 * metrics.roc_auc_score(Y, pred_prob) - 1

as you can see in the last line of code there is clearly no metric called roc_auc_score to calculate the gini.

Really appreciate any help on this.

Upvotes: 1

Views: 2188

Answers (2)

Anastasiia Goi
Anastasiia Goi

Reputation: 1

In PySpark, obtaining the ROC AUC score can be slightly different than in sklearn.

Replace the MulticlassClassificationEvaluator with BinaryClassificationEvaluator:

gbt_evaluator = BinaryClassificationEvaluator(
labelCol="label", rawPredictionCol="rawPrediction",   metricName="areaUnderROC")

Here, note the change from predictionCol to rawPredictionCol. The rawPredictionCol contains the raw prediction values which are the scores/probabilities of the positive class, which are used to compute the ROC AUC score.

Compute the Gini coefficient:

roc_auc = gbt_evaluator.evaluate(gbt_predictions)

gini = 2*roc_auc - 1

Upvotes: 0

Dani Sanchez
Dani Sanchez

Reputation: 11

Normally Gini is used to evaluate a binary classification model.

You can calculate it in pyspark in the next way:

from pyspark.ml.evaluation import BinaryClassificationEvaluator

evaluator = BinaryClassificationEvaluator()
auc = evaluator.evaluate(gbt_predictions, {evaluator.metricName: "areaUnderROC"})
gini = 2 * auc - 1.0

Upvotes: 1

Related Questions