CODEWITHSUNDEEP
opengllibgdxblendingstencil-buffer
florianbaethge
florianbaethge

Reputation: 2560

Create "Union" of two masking images in OpenGL

for a current 2D project I am rendering different objects on a scene. On top of this I render images which have a cut out part, for example a transparent circle on a black image. When moving the cut-out circle, this creates the effect that of course only the within the transparent part, the background objects are visible.

Now I want to add a second masking layer with a different transparent shape on it and create a union of these two, showing the background images underneath each of the transparent parts.

The following images show an example illustration:

Background objects Background objects

Mask 1 Masking image 1

Mask 2 Masking image 2

Desired Result Desired Result

For rendering, I am using libgdx with OpenGL 2.0 and scene2d as scenegraph. Basically, the background objects are added as actors onto a stage and then another Group-object rendering the masks.

Now I've tried by setting the Blending-function while rendering the masks, but I can't figure out if its possible to "unionize" the alpha values of each mask. But is that even possible?

I've though about using stencil buffers but I can't get this to work yet. I would be thankful if anybody could give me an approach on how to achieve this effect. Also, using stencil buffers would result in a pretty chopped of edge as the mask is either 0 or 1, correct?

Upvotes: 0

Views: 573

Answers (1)

Bartek Banachewicz
Bartek Banachewicz

Reputation: 39370

A potential approach could be to use render-to-texture and compositing manually. I'm saying "potential", because there's hardly one best way here. Using built-in blending modes can certainly have some performance gains, but it limits you to the provided blending function parameters. While certainly doable with stuff like rendering the mask to the framebuffer alpha channel, and then using that with GL_DST_ALPHA/GL_ONE_MINUS_DST_ALPHA, it gets tricky once your layout gets more complex.

Render-to-texture, OTOH, has no such drawback. You're taking the control of the entire compositing function and have the freedom to do whatever processing you wish. To elaborate a bit, the rendering would work like this:

  • Set up a texture for the objects, and render your objects to it.
  • Set up a texture for the mask - this could be e.g. one-channel 8-bit. Retarget the rendering to it, and render the mask with a shader that outputs the mask value.
  • If you want to add another mask, you can either render more stuff to the same mask texture, or create yet another one.
  • Crucially, it doesn't matter which order the above operations are done, because they're completely separate and don't impact each other; in fact, if the mask doesn't change, you don't even need to re-render it.
  • Render a full-screen quad with your compositing shader, taking those two textures as inputs (uniforms).

So, to sum up, render-to-texture is a bit more flexible in terms of the compositing operation, gives you a way to do other post-effects like motiong blur, and gives you more leeway in the order of operations. OTOH, it imposes a certain limit on the number of textures or passes, uses more memory (since you'll be keeping the intermediate textures in, as opposed to just working on one framebuffer), and might have a performance penalty.

If you decide to stick to the built-in blending, it gets a bit trickier. Typically you'll want to have alpha 0 as "no image", and 1 as "all image", but in this case it might be better to think about it as a mask, where 0 is "no mask" and 1 is "full mask". Then, the blend func for the mask could simply be GL_ONE/GL_ONE, and for the final image GL_ZERO/GL_ONE_MINUS_DST_ALPHA. That certainly restricts your ability to actually do blending and masking at the same time.

There exists a function called glBlendFuncSeparate that might make it a bit more flexible, but that's still not gonna give you as many possibilities as the method mentioned above.

Alternatively, actually learning how to set up stencil buffer would solve that specific issue, since the stencil buffer is made with specifically this use in mind. There's a lot of tutorials online, but basically it amounts to a few calls of glStencil(Op|Func|Mask), optionally with disabling the writes to the color buffer with glColorMask.

Upvotes: 1

Related Questions