Smooth time-series visualization of COG GeoTIFFs in OpenLayers with WebGL without loading delays

Question

I'm implementing a temporal visualization of water depth data using OpenLayers with COG (Cloud Optimized GeoTIFF) files (large). The user should be able to scroll horizontally to navigate through time, but I'm experiencing visible loading delays when switching between rasters.

// Simplified current implementation
const layer = new WebGLTileLayer({
  source: new GeoTIFF({
    sources: [{ url: initialUrl }],
    convertToRGB: true
  })
});

// On scroll event
function handleScroll(direction) {
  const newUrl = getUrlForTimeStep(currentIndex + direction);
  layer.getSource().setUrl(newUrl); // Causes visible loading delay
  currentIndex += direction;
}

What are best practices here? I've also tried to have active layers left and right of current time index, with opacity 0, with loaded sources. It works better, but I feel I'm missing the "right" way of solving this issue.