CODEWITHSUNDEEP
deep-learninghyperparametersoptuna
Qin Liang
Qin Liang

Reputation: 1

Cannot reproduce the result of best trial by optima through best params

This problem has been bothering me for a long time. I am using optuna for automatic parameter tuning of deep learning models, and the objective function returns the average AUC of five folds. Unable to reproduce the results corresponding to the best trial obtained by optuna. I saved the best params of the best trial to a JSON file, then read this parameter and rerun the entire model.(As below) There is nearly 3% difference.

study = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler(seed=2024))
study.optimize(objective, n_trials=1)
print("Best trial:")
trial = study.best_trial
print(f"  Cindex: {trial.value}")
print("  Best hyperparameters:")
for key, value in trial.params.items():
    print(f"    {key}: {value}")

importance = get_param_importances(study)
print("参数重要性：")
for param, score in importance.items():
    print(f"{param}: {score:.4f}")

best_params = trial.params

with open(json_results, "w") as f:
    json.dump({"C-index": trial.value, "Params": best_params}, f, indent=4)

if anyone has encountered the same problem？

I initially suspected it was randomness, but I don't think it would lead to a 3% difference. Further, I fixed all the random numbers that I can fix, including data partitioning, CUDA training, and optuna samplers, etc

Upvotes: 0

Views: 40

Answers (0)

Related Questions