Terminate a benchmark early because of long score calculation

Question

What i want to achieve

Currently i am running large inputs in my OptaPlanner project and with the current implementation of the constraints they are taking a long time even to calculate the initial score. So a given solver destroys the whole benchmark because it gets stuck and can not terminate. As a score calculation type i am using Drools.

I am trying to achieve an early termination of a solver that after a certain amount of time still has not passed the initial score calculation(no "Solving started" is displayed). So in a single benchmark i want to run multiple different inputs and for each of them i want to have a given timer and if that timer expires before the initial score calculation is done i want the solver to be terminated immediately. A desirable option would be to have a percentage of how much of the score calculation was completed.

The reason why i'm not just jumping on to doing optimizations is because i want to have a baseline for comparison and keep track of the results as the optimizations go on. So the information how much percentage of the initial score calculation has passed is vital for me.

What i have/know currently

1. The version of OptaPlanner that i'm using is the one from GitHub that has the whole source code open(it is not the official version from the website which is compiled in JAR's and the core is not editable)
2. Implemented timers for each solver of the benchmark which after a given time period call the solver.terminateEarly() method.
3. Each solver runs on it's unique thread. So the relation solver : thread is 1:1. The way i am finding out which solver is currently executing the code is by doing a lookup in a Map<Integer,Solver> solverMap where the key is the value of the hashCode of the thread executing the solver -> Thread.currentThread().hashCode(). As the solvers start and finish this Map is being updated. This way i am able to do the lookup from all the places (optaplanner-examples, optaplanner-core, optaplanner-benchmark projects and Drools rules(example below))
4. Found out kcontext.getKieRuntime().halt() from the Drools documentation which is used to terminate rule execution immediately.
5. Implemented specialized rules that will reach the then part after each change of a planning/shadow entity and from the then part checks first is the solver terminated early(by the corresponding Timer) and if it is calls kcontext.getKieRuntime().halt(). For example:

In the rule below the then part will be reached after each change in a ShiftAssignment instance and the rule execution will be stopped if the solver is set to be terminated early.

salience 1 //so that it is triggered first
rule "ShiftAssignmentChange"
    when 
        ShiftAssignment()   
    then 
    if(TerminateBenchmarkEarly.solverMap.get(Thread.currentThread().hashCode()).isTerminateEarly()){
        kcontext.getKieRuntime().halt();//This command is used to terminate the fire loop early.    
    }   
end

The intention with these rules is that they have salience 1 opposed to the default option which is 0, so they will be the first ones that will be executed and the rule execution will be immediately stopped 6. The kieSession.fireAllRules() call from the org.optaplanner.core.impl.score.director.drools.DroolsScoreDirector calculateScore method returns the number of rules that were executed. I can use this measure as a baseline for how much the initial score has achieved.As the optimizations go on it is expected that this number grows higher and the time taken is becoming smaller.

The problem that i'm facing currently

The problem that i have is that even with this implemented again it is taking it a lot of time to reach the check in the rules, or in some cases crashes because of an OutOfMemory error. Turning on the Trace option for Drools i was able to see that some smaller part of the time it was inserting the facts into the working memory, and then after that it constantly is outputting TRACE BetaNode stagedInsertWasEmpty=false. The problem lies in the kieSession.fireAllRules() call from the org.optaplanner.core.impl.score.director.drools.DroolsScoreDirector calculateScore method, the code of fireAllRules is from the Drools core and this code is compiled into a JAR so it cannot be edited.

Conclusion

Anyway i know that this is somehow a hack but as i say above i need this information as a baseline to know where my current solution is and keep track of the benchmark information as the optimizations go on. If there is different(smarter) way with which i can achieve this, i would be happy to do it.

Results from a benchmark

Input 1

• Entity count: 12,870
• Variable count: 7,515
• Maximum value count: 21
• Problem scale: 22,068
• Memory usage after loading the inputSolution (before creating the Solver): 44,830,840 bytes on average.
• Average score calculation speed after Construction Heuristic = 1965/sec
• Average score calculation speed after Local Search = 1165/sec
• Average score calculation speed after Solver is finished = 1177/sec

Input 2

• Entity count: 17,559
• Variable count: 7,515
• Maximum value count: 8
• Problem scale: 21,474
• Memory usage after loading the inputSolution (before creating the Solver): 5,964,200 bytes on average.
• Average score calculation speed after Construction Heuristic = 1048/sec
• Average score calculation speed after Local Search = 1075/sec
• Average score calculation speed after Solver is finished = 1075/sec

Input 3

• Entity count: 34,311
• Variable count: 14,751
• Maximum value count: 8
• Problem scale: 43,358
• Memory usage after loading the inputSolution (before creating the Solver): 43,178,536 bytes on average.
• Average score calculation speed after Construction Heuristic = 1134/sec
• Average score calculation speed after Local Search = 450/sec
• Average score calculation speed after Solver is finished = 452/sec

Input 4

• Entity count: 175,590
• Variable count: 75,150
• Maximum value count: 11
• Problem scale: 240,390
• Memory usage after loading the inputSolution (before creating the Solver): 36,089,240 bytes on average.
• Average score calculation speed after Construction Heuristic = 739/sec
• Average score calculation speed after Local Search = 115/sec
• Average score calculation speed after Solver is finished = 123/sec

Input 5

• Entity count: 231,000
• Variable count: 91,800
• Maximum value count: 31
• Problem scale: 360,150
• Memory usage after loading the inputSolution (before creating the Solver): 136,651,744 bytes on average.
• Average score calculation speed after Construction Heuristic = 142/sec
• Average score calculation speed after Local Search = 11/sec
• Average score calculation speed after Solver is finished = 26/sec

Input 6

• Entity count: 770,000
• Variable count: 306,000 '
• Maximum value count: 51
• Problem scale: 1,370,500
• Memory usage after loading the inputSolution (before creating the Solver): 114,488,056 bytes on average.
• Average score calculation speed after Construction Heuristic = 33/sec
• Average score calculation speed after Local Search = 1/sec
• Average score calculation speed after Solver is finished = 17/sec

When commenting out the rules in Drools i get the next average score calculation speed (for Input 6):

• After Construction Heuristic = 17800/sec
• After Local Search = 22557/sec
• After Solver is finished = 21690/sec

Answer 1

If possible, I'd first focus on making the DRL faster, instead of these hacks. So that comes down to figuring out which score rules are slow. Use the score calculation speed (in the last INFO log line) to determine that by commenting out score rules and seeing their impact on the score calculation speed.

That being said, normally I'd advice to look at unimprovedSecondsSpentLimit or a custom Termination - but that indeed won't help as those aren't checked while the initial score is calculated from scratched: they are only checked between every move (so between every fireAllRules(), usually 10k/sec).