CODEWITHSUNDEEP
pythonpandash2ogbm
J. Lee
J. Lee

Reputation: 21

Unable to reproduce H2O GBM predictions despite setting seed

I'm trying to run multiple H2O models on different response variables in a for loop.

H2O cluster uptime:         53 mins 11 secs
H2O cluster timezone:       Etc/UTC
H2O data parsing timezone:  UTC
H2O cluster version:        3.22.1.1
H2O cluster version age:    2 hours and 15 minutes
H2O cluster name:           H2O_from_python_root_np3l2m
H2O cluster total nodes:    1
H2O cluster free memory:    13.01 Gb
H2O cluster total cores:    8
H2O cluster allowed cores:  8
H2O cluster status:         locked, healthy
H2O connection url:         http://localhost:54321
H2O connection proxy:
H2O internal security:      False
H2O API Extensions:         XGBoost, Algos, AutoML, Core V3, Core V4
Python version:             2.7.12 final

I have set a seed for selecting the train/validation sets and the model itself. I have early stopping active but according to the documentation results should be reproducible as long as score_tree_interval is active.

### This is the code that's defining the model

def append_probs(hframe, response_col, model):
  pd_df = h2o.as_list(hframe).copy()
  pd_df.loc[:,'pred'] = h2o.as_list(model.predict(hframe)).values
  pd_df.loc[:,'error'] = pd_df['pred'] - pd_df[response_col]
  return pd_df

def run_model(response_col, model_typ, hframe_train, hframe_pred):
  h2o_dtypes = [hframe_train.type(e) for e in hframe_train.columns]
  data = h2o.deep_copy(hframe_train,'data')
  mapping = {'new_email_ldsub':'live_pp',
             'new_call_ldsub':'live_pp',
             'used_email_ldsub':'live_usedplus',
             'used_call_ldsub':'live_usedplus',
             'myapp_edm_ldsub':'live_myapp',
             'cc_edm_ldsub':'live_cc',
             'fbm_call_ldsub':'live_fbm',
             'fbm_email_ldsub':'live_fbm'}
  data = data[data[mapping[response_col]]==1]

  train, valid = data.split_frame([0.8], seed=1234)

  X = hframe_train.col_names[:-14]
  print X
  y = response_col
  print y

  if model_typ == 'gbm':
    model = H2OGradientBoostingEstimator(
      ntrees=512,
      learn_rate=0.08,
      max_depth=7,
      col_sample_rate = 0.7,
      sample_rate = 0.9,
      stopping_tolerance=1e-05,
      stopping_rounds=2,
      score_tree_interval=5,
      #nfolds=5,
      #fold_assignment = "Random",
      distribution = 'poisson',
      seed=20000,
      stopping_metric='mae',
      min_rows = 10,
      nbins = 30

  model.train(X, y, training_frame=train, validation_frame=valid)

  pred_df = append_probs(hframe_pred,response_col,model)

  return model, pred_df

### This is the code that runs the model

gbm_results = pd.DataFrame()

gbm_mapping = {'live_pp':['new_call_ldsub','new_email_ldsub'],
           'live_usedplus':['used_call_ldsub','used_email_ldsub'],
           'live_myapp':['myapp_edm_ldsub'],
           'live_cc':['cc_edm_ldsub'],
           'live_fbm':['fbm_call_ldsub','fbm_email_ldsub']}

gbm_train_err = {}
gbm_valid_err = {}
gbm_xval_err = {}


for k,v in gbm_mapping.iteritems():
  for e in v:
    gbm_mod, gbm_pred_df = run_model(e,'gbm',hframe,hframe_forecast_pred)
    gbm_pred_df = gbm_pred_df[['id','month','pred']]
    gbm_pred_df = gbm_pred_df.groupby(['id','month'])['pred'].sum().reset_index()
    gbm_pred_df.loc[:,'product'] = str(e)
    gbm_train_err[str(e)] = [gbm_mod.mae(train=True),gbm_mod.rmse(train=True)]
    gbm_valid_err[str(e)] = [gbm_mod.mae(valid=True),gbm_mod.rmse(valid=True)]
    gbm_xval_err[str(e)] = [gbm_mod.mae(xval=True),gbm_mod.rmse(xval=True)]
    gbm_results = pd.concat([gbm_results, gbm_pred_df])

gbm_results['process_month'] = pd.to_datetime(gbm_results['process_month'],unit='ms')

Based on the documentation I'm expecting the results for each model to be reproducible/close.

Upvotes: 2

Views: 1007

Answers (1)

Lauren
Lauren

Reputation: 5778

As of the latest version of H2O-3 3.22.1.1 the reproducibility requirements are listed in the documentation here.

For your convenience, here are the requirements for reproducibility of a model on a single node:

Note that in addition to a seed, you need to use the same data (same splits), same parameters, and either don't use early stopping or use early stopping with score_tree_interval set.

How to guarantee reproducibility in single node cluster?

The following criteria must be met to guarantee reproducibility in a single node cluster:

  • same training data

Note: If you have H2O import a whole directory with multiple files instead of a single file, we do not guarantee reproducibility because the data may be shuffled during import.

  • same parameters used to train the model
  • same seed set (this is required when any sampling is done)
  • no early stopping or early stopping with score_tree_interval set and same validation data

Upvotes: 2

Related Questions