CODEWITHSUNDEEP
pythonapache-sparkmachine-learningpyspark
Larissa Leite
Larissa Leite

Reputation: 1368

pyspark.sql.utils.IllegalArgumentException: 'requirement failed: Invalid initial capacity'

I am trying to run a cross validation with a decision tree in Spark using the ML library but I am getting this error when calling cv.fit(train_dataset):

pyspark.sql.utils.IllegalArgumentException: u'requirement failed: Invalid initial capacity'

I haven't found much information about what it could be other than the dataframe being empty but it isn't. This is my code:

df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data')
df.columns = ['Sex', 'Length', 'Diameter', 'Height', 'Whole weight', 'Schuked weight', 'Viscera weight', 'Shell weight', 'Rings']
train_dataset = sqlContext.createDataFrame(df)

column_types = train_dataset.dtypes

categoricalCols = []
numericCols = []

for ct in column_types:
    if ct[1] == 'string':
        categoricalCols += [ct[0]]
    else:
        numericCols += [ct[0]]

stages = []
for categoricalCol in categoricalCols:
    stringIndexer = StringIndexer(inputCol=categoricalCol, outputCol=categoricalCol+"Index")
    stages += [stringIndexer]

assemblerInputs = map(lambda c: c + "Index", categoricalCols) + numericCols
assembler = VectorAssembler(inputCols=assemblerInputs, outputCol="features")
stages += [assembler]

labelIndexer = StringIndexer(inputCol='Rings', outputCol='indexedLabel')
stages += [labelIndexer]

dt = DecisionTreeClassifier(labelCol="indexedLabel", featuresCol="features")

evaluator = MulticlassClassificationEvaluator(labelCol='indexedLabel', predictionCol='prediction', metricName='f1')

paramGrid = (ParamGridBuilder()
             .addGrid(dt.maxDepth, [1,2,6])
             .addGrid(dt.maxBins, [20,40])
             .build())

stages += [dt]
pipeline = Pipeline(stages=stages)

cv = CrossValidator(estimator=pipeline, estimatorParamMaps=paramGrid, evaluator=evaluator, numFolds=1)

cvModel = cv.fit(train_dataset)
train_dataset = cvModel.transform(train_dataset)

I am running Spark standalone locally. What could be wrong?

Thanks!

Upvotes: 0

Views: 3231

Answers (1)

Larissa Leite
Larissa Leite

Reputation: 1368

So, the problem was setting the numFolds parameter of the CrossValidation to 1. If I wanna do the parameter tuning with ParamGrid with only one single train-test split, apparently I need to use TrainValidationSplit instead.

Upvotes: 1

Related Questions