Optaplanner cloning solution between afterVariableChanged and calculateScore

Question

We are using OptaPlanner with the IncrementalScoreCalculator model and were working under the assumption that intermediate clones (for saving a new best solution when encontered) were made just after the calculateScore call (therefore using the newly calculated hard/soft score in order to determine whether it is the best solution).

As we are working with a problem state that is only updated in the calculateScore phase (due to performance reasons) this leads to the selection of a new solution with an out of date score.

Is there any way we can configure the best solution to be selected/cloned only just after the calculateScore step?

Update: Running the solver with the FULL_ASSERT directive avoids the issue : cloning always happens after calculateScore(). However, if no directive is used the following happens (each line represents a Solution Id - Iteration key):

2022-10-20 01:05:02.828 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1000 - AfterVariableChanged

2022-10-20 01:05:02.828 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1000 - CalculateScore

2022-10-20 01:05:02.829 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - BeforeVariableChanged

2022-10-20 01:05:02.829 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - AfterVariableChanged

2022-10-20 01:05:02.829 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - BeforeVariableChanged

2022-10-20 01:05:02.829 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - AfterVariableChanged

2022-10-20 01:05:02.829 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - BeforeVariableChanged

2022-10-20 01:05:02.829 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - AfterVariableChanged

2022-10-20 01:05:02.830 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - BeforeVariableChanged

2022-10-20 01:05:02.830 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - AfterVariableChanged

2022-10-20 01:05:02.830 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - BeforeVariableChanged

2022-10-20 01:05:02.830 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - AfterVariableChanged

2022-10-20 01:05:02.830 INFO 86000 --- [nio-5000-exec-5] c.o.p.o.h.s.SNPSchedulingSolutionCloner : 4-1001 - CloneSolution -- here is where the solution is cloned, before a new scoreCalculation and in the same thread --

2022-10-20 01:05:03.089 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - BeforeVariableChanged

2022-10-20 01:05:03.089 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - AfterVariableChanged

2022-10-20 01:05:03.089 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - BeforeVariableChanged

2022-10-20 01:05:03.089 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - AfterVariableChanged

2022-10-20 01:05:03.089 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - BeforeVariableChanged

2022-10-20 01:05:03.089 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - AfterVariableChanged

2022-10-20 01:05:03.089 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1001 - CalculateScore

2022-10-20 01:05:03.263 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1002 - BeforeVariableChanged

2022-10-20 01:05:03.263 INFO 86000 --- [nio-5000-exec-5] .SNPSchedulingIncrementalScoreCalculator : 4-1002 - AfterVariableChanged

We have tested with OptaPlanner 8.29.0.Final and 8.16.1.Final.

Evaluating the BestSolutionRecaller method this is what I've found: A clone is created as the method understands that the current score (0 hard / 7 soft, from iteration 1000) is better than the best score from SolverScope (0 hard / 0 soft). However, at a prior iteration (741) a best score of (0 hard / 7 soft) had already been found.

Answer 1

From the sequence of operations you posted, I do not (yet) see an issue. The fact that there are multiple variable change events before a score is calculated could be totally normal; a move that switches two vehicle destinations does two variable updates, a pillar move that changes all beds in the same department could possibly update dozens of variables.

It is logical that score calculation would only be called after all of those updates have been triggered, as anywhere in the middle would be an inconsistent solution state. Score calculation must happen after a move finishes, not after events are triggered.

Furthermore, this code is exercised by our users daily, it is also heavily covered with tests. It is the very core of OptaPlanner, it has to work. If we had a bug in here, we would have been notified immediately, and not months after the fact. (I assume your mention of 8.16.1.Final means that you are seeing the issue there as well. That release is, by now, nearly a year old.)

All of the above leads me to conclude that there is no issue on OptaPlanner side. Running NON_INTRUSIVE_FULL_ASSERT environment mode can either point you to an issue in your score calculator implementation, or prove me wrong. That said, if the assert mode passes, I'm going to have to see proof that, when the move finishes, there really is no score calculation. This would be a serious bug, and that would be the answer to your question.