Potentially biased sampling in Tree-Parzen optimization from Hyperopt

Question

I'm using the Tree-Parzen algo implementation from Hyperopt for blackbox optimization task that I have and I'm consistently observing an odd phenomenon.

The plot below displays the phenomenon. You can see that the sampler is focusing on a low-loss region (the dark ball near the top right). This is normal, indeed expected.

What I find odd is that the sampler doesn't seem to be exploring at all from the regions parallel to this point. You'll see that there are "channels" that are mostly un-sampled perpendicular to the identified low loss region.

The plot only shows two of 4 parameters that are being optimized for, but the phenomenon is observable with all pairs of parameters.

This seems to me to be non-optimal behavior. While we want the search algo to focus on low loss regions to improve efficiency, we also want it to sufficiently explore nearby regions to ensure robustness of the results.

To improve exploration, I've set the first half of the samples to be suggested at random, and the last half to be suggested by the TPE algo.

The chart below is from this hybrid setup, i.e. increasing initial exploration didn't seem to improve exploration in these nearby zones.

I would like, first, to better understand why this happens, and second, to improve exploration.

Any thoughts and suggestions are much appreciated.

Thanks!