Pyspark: IllegalArgumentException: u'requirement failed: init value should <= bound'

Question

This error has pretty much come out of no where. I'm running a simple linear regression in pysark using the following code:

from pyspark.ml.regression import LinearRegression

# initialize regression model object
linreg = LinearRegression()

# Fit 2 models, using different regularization parameters
modelA = linreg.fit(X_train_vector)

preds = modelA.transform(X_test_vector)

from pyspark.ml.evaluation import RegressionEvaluator
evaluator = RegressionEvaluator(metricName="r2")
R2 = evaluator.evaluate(preds)
print("ModelA: R2 = " + str(R2))

My first thought was that there could be null values, but I've already taken care of that prior to creating my feature set.

Here is the full error:

IllegalArgumentException: u'requirement failed: init value should <= bound'
---------------------------------------------------------------------------
IllegalArgumentException                  Traceback (most recent call last)
<command-37272> in <module>()
      5 
      6 # Fit 2 models, using different regularization parameters
----> 7 modelA = linreg.fit(X_train_vector)
      8 
      9 preds = modelA.transform(X_test_vector)

/databricks/spark/python/pyspark/ml/base.pyc in fit(self, dataset, params)
     62                 return self.copy(params)._fit(dataset)
     63             else:
---> 64                 return self._fit(dataset)
     65         else:
     66             raise ValueError("Params must be either a param map or a list/tuple of param maps, "

/databricks/spark/python/pyspark/ml/wrapper.pyc in _fit(self, dataset)
    263 
    264     def _fit(self, dataset):
--> 265         java_model = self._fit_java(dataset)
    266         return self._create_model(java_model)

I'm not entirely sure what this means tbh.

Answer 1

It looks like you are using outdated Spark version and experience SPARK-21523 - Fix bug of strong wolfe linesearch init parameter lose effectiveness.

If you use Spark 2.2 make sure to update to 2.2.1 release, where the bug is already fixed.