Masking with glBlendFunc, without frame buffers

Question

I'm hoping this is possible to do without using frame buffers or shaders, just by straight up using the glBlendFunc or glBlendFuncSeparate.

I'm rendering my scene normally with my standard blend mode:

glBlendFunc(GL.GL_SRC_ALPHA, GL.GL_ONE_MINUS_SRC_ALPHA);

Ontop of that scene, I want to draw a texture which is masked by some other texture. These are drawn at arbitrary positions (not necessarily rectangular, not necessarily the same size / position as each other).

The order is; render the masked texture, then the mask texture.

The masked texture is a regular image, with alpha.

The mask texture either black RGBA(0, 0, 0, 255), or transparent RGBA(0, 0, 0, 0)

I want anything that the black does NOT touch, to be invisible. Basically, the final result should be:

RGBA(masked.r, masked.g, masked.b, masked.a * mask.a)

Below are images of the ordering, to explain what I mean. I'm really looking for a solution to avoid having to use a different shader or stick things onto a framebuffer. If it absolutely isn't possible, please let me know.

Answer 1

I'll explain why this isn't possible. Masking with blending requires three passes because it has three parts: the destination, the source, and the mask. No matter what you do, you must blend the source and the mask into a framebuffer and THEN render to destination.

The stencil buffer, however, is built into the default window framebuffer, provided you tell OpenGL to provide for it (like you would a depth buffer), and appears to do exactly what you want. As a GLUT call, it would look like this to initialize the stencil buffer in your window along with the alpha-enabled color and depth buffers, in a double-buffered window:

glutInitDisplayMode(GLUT_DOUBLE | GLUT_RGBA | GLUT_ALPHA | GLUT_DEPTH | GLUT_STENCIL);

The stencil buffer is able to do exactly what you need - you can draw a shape in it, and selectively tell it to either discard or keep pixels inside that shape. Here's an example of how to use it, modified from the the OpenGL Red Book:

GLdouble dRadius = 0.1; // Initial radius of spiral
GLdouble dAngle;        // Looping variable

// Use 0 for clear stencil, enable stencil test
glClearStencil(0);
glEnable(GL_STENCIL_TEST);

// Clear stencil buffer
glClear(GL_STENCIL_BUFFER_BIT);

// All drawing commands fail the stencil test, and are not
// drawn, but increment the value in the stencil buffer.
// glStencilFunc takes three arguments: the stencil function, the reference value, and the mask value. Whenever the stencil function is tested (for example GL_LESS), both the reference and the stencil value being tested from the buffer is bitwise ANDed with the mask: GL_LESS returns true if (ref & mask) < (stencil & mask).
glStencilFunc(GL_NEVER, 0x0, 0x0);
// glStencilOp takes three arguments: the stencil operation to call when the stencil test fails, the stencil operation to call when the stencil test passes but the depth test fails, and the stenciloperation to call when the stencil test passes AND the depth test passes (or depth test is disabled or no depth buffer is allocated).
glStencilOp(GL_INCR, GL_INCR, GL_INCR);

// Spiral pattern will create stencil pattern
// Draw the spiral pattern with white lines. We
// make the lines  white to demonstrate that the
// stencil function prevents them from being drawn
glColor3f(1.0f, 1.0f, 1.0f);
glBegin(GL_LINE_STRIP);
    for(dAngle = 0; dAngle < 400.0; dAngle += 0.1)
        {
        glVertex2d(dRadius * cos(dAngle), dRadius * sin(dAngle));
        dRadius *= 1.002;
        }
glEnd();

// Now, allow drawing, except where the stencil pattern is 0x1
// and do not make any further changes to the stencil buffer
glStencilFunc(GL_NOTEQUAL, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

// Now draw red square
glColor3f(0.0f, 1.0f, 0.0f);
glRectf(0, 0, 200, 200);

The output of this drawing code is a red square, with a spiral across it, starting at (1, 1). The spiral is made up of discarded pixels, and as such will be the same color as the cleared color buffer. If you were to use this code for your purposes, you would draw the square where you wanted your texture to be where the spiral code is written, and then use GL_EQUAL as the stencil function, drawing your masked texture where the red square is drawn. More information on the stencil buffer can be found here.