CODEWITHSUNDEEP
c++openglgraphicsstencil-buffer
zyneragetro
zyneragetro

Reputation: 149

OpenGL - Stencil Test in a Rectangle

I am planning to draw a rectangle with a hole in its center. I am trying to use the stencil test on it but I can't make it work. You can see below for how I do it.

glEnable(GL_STENCIL_TEST);
glColorMask(GL_FALSE,GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);
glStencilFunc(GL_ALWAYS, 2, ~0);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
// Draw the rectangle here
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 2, ~0);
glStencilOp(GL_INCR, GL_INCR, GL_INCR);

What am I doing wrong here? Any help would much be appreciated! :)

Upvotes: 1

Views: 1443

Answers (1)

Angew is no longer proud of SO
Angew is no longer proud of SO

Reputation: 171117

I assume your situation is that you already have something drawn in the framebuffer, and now you want to draw a rectangle with a hole in it, so that it does not cover what's underneath the hole, but covers what't underneath the non-hole part.

Logically, this means you first draw the hole into the stencil buffer, and then use the stencil test to exclude those fragments when drawing the rectangle.

In code, it looks something like this:

glEnable(GL_STENCIL_TEST);

// Fill stencil buffer with 0's
glClearStencil(0);
glClear(GL_STENCIL_BUFFER_BIT);

// Write 1's into stencil buffer where the hole will be
glColorMask(GL_FALSE,GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);
glStencilFunc(GL_ALWAYS, 1, ~0);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
drawHoleShape();

// Draw rectangle, masking out fragments with 1's in the stencil buffer
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);
glStencilFunc(GL_NOTEQUAL, 1, ~0);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
drawRectangle();

// Cleanup, if necessary
glDisable(GL_STENCIL_TEST);

Of course, you can use 2 (or any other stencil bit/value) instead of 1.

Upvotes: 3

Related Questions