iOS10 + SceneKit: render a video with custom shader

Question

How can I render a video as a diffuse texture on a node in SceneKit while using custom shader? This sounds like a simple thing to do but:

• SceneKit elements don't support AVPlayerLayer as input texture (SO)
• alternatives are:
  • using a SpriteKit scene with a SKVideoNode but:
    • Since iOS 10, rendering with openGL fails as soon as a SpriteKit scene is involved (SO) I've filled a radar
    • I could not figure out how to pass uniforms to a shader modifier when using Metal (SO) Things were working perfectly with openGL (on iOS 9, when SpriteKit was working) There's surprisingly very little documentation out there on that.
    • SCNProgram seems to fails as soon as a video is involved in the SceneKit scene (SO) Things work fine with the same setup without video
  • transforming the video output to a CGImage, passing it to a CALayer, and passing this layer to the SceneKit video node (no SpriteKit involved) This is detailed here, but not a viable solution as soon as the video is a bit large, because all the output is getting copied to image and this is costly..

I'm still quite reluctant to get out of SceneKit, since it's handling a lot of the annoying low level things with opengl/metal, but I can't get things to work right now...

As an additional bad news, it looks like SpriteKit has memory leak when used with Metal, and that it keeps alive the SceneView, and therefore all the parent views and everything attached to them...

You can find a reproduction of this issue here, and a simpler one here:

Answer 1

Not being able to solve this issue, I wrote my own library that mimics some functionalities of SceneKit: https://github.com/team-pie/DDDKit