MTKView refresh issue

Question

I am compositing an array of UIImages via an MTKView, and I am seeing refresh issues that only manifest themselves during the composite phase, but which go away as soon as I interact with the app. In other words, the composites are working as expected, but their appearance on-screen looks glitchy until I force a refresh by zooming in/translating, etc.

I posted two videos that show the problem in action: Glitch1, Glitch2

The composite approach I've chosen is that I convert each UIImage into an MTLTexture which I submit to a render buffer set to ".load" which renders a poly with this texture on it, and I repeat the process for each image in the UIImage array.

The composites work, but the screen feedback, as you can see from the videos is very glitchy.

Any ideas as to what might be happening? Any suggestions would be appreciated

Some pertinent code:

for strokeDataCurrent in strokeDataArray {

        let strokeImage = UIImage(data: strokeDataCurrent.image)
        let strokeBbox = strokeDataCurrent.bbox
        let strokeType = strokeDataCurrent.strokeType
        self.brushStrokeMetal.drawStrokeImage(paintingViewMetal: self.canvasMetalViewPainting, strokeImage: strokeImage!, strokeBbox: strokeBbox, strokeType: strokeType)

} // end of for strokeDataCurrent in strokeDataArray

...

func drawStrokeUIImage (strokeUIImage: UIImage, strokeBbox: CGRect, strokeType: brushTypeMode) {

    // set up proper compositing mode fragmentFunction
    self.updateRenderPipeline(stampCompStyle: drawStampCompMode)

    let stampTexture = UIImageToMTLTexture(strokeUIImage: strokeUIImage)
    let stampColor = UIColor.white
    let stampCorners = self.stampSetVerticesFromBbox(bbox: strokeBbox)
    self.stampAppendToVertexBuffer(stampUse: stampUseMode.strokeBezier, stampCorners: stampCorners, stampColor: stampColor)
    self.renderStampSingle(stampTexture: stampTexture)


  } // end of func drawStrokeUIImage (strokeUIImage: UIImage, strokeBbox: CGRect)

func renderStampSingle(stampTexture: MTLTexture) {

    // this routine is designed to update metalDrawableTextureComposite one stroke at a time, taking into account
    // whatever compMode the stroke requires. Note that we copy the contents of metalDrawableTextureComposite to
    // self.currentDrawable!.texture because the goal will be to eventually display a resulting composite

    let renderPassDescriptorSingleStamp: MTLRenderPassDescriptor? = self.currentRenderPassDescriptor

    renderPassDescriptorSingleStamp?.colorAttachments[0].loadAction = .load
    renderPassDescriptorSingleStamp?.colorAttachments[0].clearColor = MTLClearColorMake(0, 0, 0, 0)
    renderPassDescriptorSingleStamp?.colorAttachments[0].texture = metalDrawableTextureComposite 

    // Create a new command buffer for each tessellation pass
    let commandBuffer: MTLCommandBuffer? = commandQueue.makeCommandBuffer()

    let renderCommandEncoder: MTLRenderCommandEncoder? = commandBuffer?.makeRenderCommandEncoder(descriptor: renderPassDescriptorSingleStamp!)

    renderCommandEncoder?.label = "Render Command Encoder"
    renderCommandEncoder?.setTriangleFillMode(.fill)
    defineCommandEncoder(
      renderCommandEncoder: renderCommandEncoder,
      vertexArrayStamps: vertexArrayStrokeStamps,
      metalTexture: stampTexture) // foreground sub-curve chunk

    renderCommandEncoder?.endEncoding() // finalize renderEncoder set up

    //begin presentsWithTransaction approach  (needed to better synchronize with Core Image scheduling
    copyTexture(buffer: commandBuffer!, from: metalDrawableTextureComposite, to: self.currentDrawable!.texture)
    commandBuffer?.commit() // commit and send task to gpu

    commandBuffer?.waitUntilScheduled()
    self.currentDrawable!.present()
    // end presentsWithTransaction approach
    self.initializeStampArray(stampUse: stampUseMode.strokeBezier) // clears out the stamp array in preparation of next draw call

  } // end of func renderStampSingle(stampTexture: MTLTexture)

Answer 1

First of all, the domain Metal is very deep, and it's use within the MTKView construct is sparsely documented, especially for any applications that fall outside the more traditional gaming paradigm. This is where I have found myself in the limited experience I have accumulated with Metal with the help from folks like @warrenm, @ken-thomases, and @modj, whose contributions have been so valuable to me, and to the Swift/Metal community at large. So a deep thank you to all of you.

Secondly, to anyone troubleshooting metal, please take note of the following: If you are getting the message:

[CAMetalLayerDrawable present] should not be called after already presenting this drawable. Get a nextDrawable instead

please don't ignore it. It mays seem harmless enough, especially if it only gets reported once. But beware that this is a sign that a part of your implementation is flawed, and must be addressed before you can troubleshoot any other Metal-related aspect of your app. At least this was the case for me. As you can see from the video posts, the symptoms of having this problem were pretty severe and caused unpredictable behavior that I was having a difficult time pinpointing the source of. The thing that was especially difficult for me to see was that I only got this message ONCE early on in the app cycle, but that single instance was enough to throw everything else graphically out of whack in ways that I thought were attributable to CoreImage and/or other totally unrelated design choices I had made.

So, how did I get rid of this warning? Well, in my case, I assumed that having the settings:

self.enableSetNeedsDisplay = true // needed so we can call setNeedsDisplay()  to force a display update as soon as metal deems possible
self.isPaused = true // needed so the draw() loop does not get called once/fps
self.presentsWithTransaction = true // for better synchronization with CoreImage (such as simultaneously turning on a layer while also clearing MTKView)

meant that I could pretty much call currentDrawable!.present() or commandBuffer.presentDrawable(view.currentDrawable) directly whenever I wanted to refresh the screen. Well, this is not the case AT ALL. It turns out these calls should only be made within the draw() loop and only accessed via a setNeedsDisplay() call. Once I made this change, I was well on my way to solving my refresh riddle.

Furthermore, I found that the MTKView setting self.isPaused = true (so that I could make setNeedsDisplay() calls directly) still resulted in some unexpected behavior. So, instead, I settled for:

self.enableSetNeedsDisplay = false // needed so we can call setNeedsDisplay()  to force a display update as soon as metal deems possible
self.isPaused = false // draw() loop gets called once/fps
self.presentsWithTransaction = true // for better synchronization with CoreImage

as well as modifying my draw() loop to drive what kind of update to carry out once I set a metalDrawableDriver flag AND call setNeedsDisplay():

override func draw(_ rect: CGRect) {

autoreleasepool(invoking: { () -> () in

  switch metalDrawableDriver {

  case stampRenderMode.canvasRenderNoVisualUpdates:

    return

  case stampRenderMode.canvasRenderClearAll:

    renderClearCanvas()

  case stampRenderMode.canvasRenderPreComputedComposite:

    renderPreComputedComposite()

  case stampRenderMode.canvasRenderStampArraySubCurve:
      renderSubCurveArray()

  } // end of switch metalDrawableDriver

}) // end of autoreleasepool

} // end of draw()

This may seem round-about, but it was the only mechanism I found to get consistent user-driven display updates.

It is my hope that this post describes an error-free and viable solution that Metal developers may find useful in the future.