Openseadragon custom tile source dimensions result into strange behavior

Question

I've been working with Openseadragon with Angular and DeepzoomImage with a Rust backend to be generated.

I have an CustomViewer that has a CustomTileSource which accepts the hight and width as the level 0 dimensions (highest resolution):

export class CustomTileSource extends OpenSeadragon.TileSource {
  /**
   * Create a new CustomTileSource.
   * 
   */
  constructor(port: number, width: number, height: number) { // width and height from backend
    super({
      width: width,
      height: height,
      tileSize: 254,
      minLevel: 0,
      maxLevel: 15,
    });
  }
  override getTileUrl(level: number, x: number, y: number) {
    // async call to Rust backend for rendering tile
    return `${this.renderingApiHost}:${this.port}/tile/${level}/${x}/${y}`;
  }
}

I use it to create a viewer this way:

    var viewer = OpenSeadragon({
      element: osdViewerElement.nativeElement,
      prefixUrl: '//openseadragon.github.io/openseadragon/images/',
      // homeFillsViewer: true,
      // Custom tile source
      tileSources: new CustomTileSource(slideMetadata.port, slideMetadata.width, slideMetadata.height),
      // Custom buttons
      zoomInButton: 'zoom-in',
      zoomOutButton: 'zoom-out',
      // Custom navigator
      showNavigator: true,
      navigatorPosition: 'TOP_RIGHT',
      navigatorHeight: '120px',
      navigatorWidth: '145px',
      gestureSettingsMouse: {
        dragToPan: false,
      }
    });

The rendering works perfect when I zoomin. However, the image below shows what's wrong with my osd viewer. I have to zoom to the area that's bounded with my red lines to get the correct behavior. The blurry tiles outsides the bounds of my lines always remain when I zoom out. What I tried so far, is to divide both width and height with random numbers, weirdly enough 4 worked like a charm. I believe the scale factor is what I am missing here, but I am not really sure.

What I tried so far, is to divide both width and height with random numbers, weirdly enough 4 worked like a charm. I believe the scale factor is what I am missing here, but I am not really sure.