Multi-pass accumulative rendering in three.js

Question

I have written code to render a scene containing lights that work like projectors. But there are a lot of them (more than can be represented with a fixed number of lights in a single shader). So right now I render this by creating a custom shader and rendering the scene once for each light. Each pass the fragment shader determines the contribution for that light and the blender adds in that contribution to the backbuffer. I found this really awkward to set up in three.js. I couldn't find a way of doing multiple passes like this where there were different materials and different geometry required for the different passes. I had to do this by having multiple scenes. The problem there is that I can't have an object3d that is in multiple scenes (Please correct me if I'm wrong). So I need to create duplicates of the objects - one for each scene it is in. This all starts looking really hacky quickly. It's all so special that it seems to be incompatible with various three.js framework features such as VR Rendering. Each light requires shadowing, but I don't have memory for a shadow buffer for each light, so it alternates between rendering the shadow buffer for the light, then the accumulation phase for that light, then the shadow buffer for the next light, then the accumulator the the next light, etc.

I'd much rather set this up in a more "three.js" way. I seem to be writing hack upon hack to get this working, each time forgoing yet another three.js framework feature that doesn't working properly in conjunction with my multi-pass technique. But it doesn't seem like what I'm doing is so out of the ordinary.

My main surprise is that I can't figure out a way to set up a multi-pass scene that does this back and forth rendering and accumulating. And my second surprise is that the Object3D's that I create don't like being added to multiple scenes a the same time, so I have to create duplicates of each object for each scene it wants to be in, in order to keep their states from interfering with each other.

So is there a way of rendering this kind of multi-pass accumulative scene in a better way? Again, I would describe it as a scene with > the max number of lights allows in a single shader pass so their contributions need to be alternatively rendered (shadow buffers) and then additively accumulated in multiple passes. The lights work like typical movie projectors that project an image (as opposed to being a uniform solid color light source).

How can I do multi-pass rendering like this and still take advantage of good framework stuff like stereo rendering for VR and automatic shadow buffer creation?

Here's a simplified snippet that demonstrates the scenes that are involved:

  renderer.render(this.environment.scene, camera, null);
  for (let i = 0, ii = this.projectors.length; i < ii; ++i) {
    let projector = this.projectors[i];
    renderer.setClearColor(0x000000, 1);
    renderer.clearTarget(this.shadowRenderTarget, true, true, false);
    renderer.render(projector.object3D.depthScene, projector.object3D.depthCamera, this.shadowRenderTarget);
    renderer.render(projector.object3D.scene, camera);
  }

  renderer.render(this.foreground.scene, camera, null);

There is a scene that renders lighting from the environment (done with normal lighting) then there is a scene per projector that computes the shadow map for the projector and then adds in the light contribution from each projector, then there is a "foreground" scene with overlays and UI stuff in it.

Is there a more "three.js" way?

Answer 1

Unfortunately i think the answer is no.

I'd much rather set this up in a more "three.js" way. I seem to be writing hack upon hack to get this working,

and welcome to the world of three.js development :)

scene graph

You cannot have a node belong to multiple parents. I believe three also does not allow you to do this:

const myPos = new THREE.Vector3()

myMesh_layer0.position = myPos
myMesh_layer1.position = myPos

It wont work with eulers, quaternions or a matrix.

Managing the matrix updates in multiple scenes would be tricky as well.

the three.js way

There is no way to go about the "hack upon hack" unless you start hacking the core.

Notice that it's 2018 but the official way of including three.js into your web app is through <src> tags.

This is a great example of where it would probably be a better idea not to do things the three.js way but the modern javascript way ie use imports, npm installs etc.

Three.js also does not have a robust core that allows you to be flexible with code around it. It's quite obfuscated and conflated with a limited number of hooks exposed that would allow you to write effects such as you want.

Three is often conflated with it's examples, if you pick a random one, it will be written in a three.js way, but far from what best javascript/coding practices are, at least today.

You'll often find large monolithic files, that would benefit from being broken up.

I believe it's still impossible to import the examples as modules.

Look at the material extensions examples and consider if you would want to apply that pattern in your project.

You can probably encounter more pain points, but this is enough to illustrate that the three.js way may not always be desirable.

remedies

Are few and far between. I've spent more than a year trying to push the onBeforeRender and onAfterRender hooks. It seems useful and allowed for some refactors, but another feature had to be nuked first.

The other feature was iterated on during the course of that year and only addressed a single example, until it was made obvious that onBeforeRender would address both the example, and allow for much more.

This unfortunately also seems to be the three.js way. Since the base is so big and includes so many examples, it's more likely that someone would try to optimize a single example, then try to find a common pattern for refactoring a whole bunch of them.

You could go and file an issue on github, but it would be very hard to argue for something as generic as this. You'd most likely have to write some code as a proposal.

This can become taxing quite quick, because it can be rejected, ignored, or you could be asked to provide examples or refactor existing ones.

You mentioned your hacks failing to work with various three's features, like VR. This i think is a problem with three, VR has been the focus of development for the past couple of years at least, but without ever addressing the core issues.

The good news is, three is more modular than it was ever before, so you can fork the project and tweak the parts you need. The issues with three than may move to a higher level, if you find some coupling in the renderer for example that makes it hard to sync your fork, it would be easier to explain than the whole goal of your particular project.