OpenGL ES 2.0 iOS - draw a rectangle into stencil buffer and limit drawing only inside it

Question

Do a good deed and help get someone (me) out of their misery, since it's New Year's Eve soon. I'm working on an iOS app, a coloring book for kids and I haven't stumbled upon OpenGL before (more precisely OpenGLES 2.0) so there's a big chance there's stuff I don't actually get in my code.

One of the tasks is to not let the brush spill out of the contour in which the user started drawing.

After reading and understanding some OpenGL basics, I found that using the stencil buffer is the right solution. This is my stencil buffer setup:

glClearStencil(0);

//clear the stencil
glClear(GL_STENCIL_BUFFER_BIT);

//disable writing to color buffer
glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );

//disable depth buffer
glDisable(GL_DEPTH_TEST);

//enable writing to stencil buffer
glEnable(GL_STENCIL_TEST);

glStencilFunc(GL_NEVER, 1, 0xFF);

glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);

[self drawStencil];

//re-enable color buffer
glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE );

//only draw where there is a 1
glStencilFunc(GL_EQUAL, 1, 1);

//keep the pixels in the stencil buffer
glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );

Right now, I'm just trying to draw a square in the stencil buffer and see if I can limit my drawing only to that square. This is the method drawing the square:

- (void)drawStencil
{
// Create a renderbuffer
GLuint renderbuffer;
glGenRenderbuffers(1, &renderbuffer);
glBindRenderbuffer(GL_RENDERBUFFER, renderbuffer);
[context renderbufferStorage:GL_RENDERBUFFER fromDrawable:(CAEAGLLayer*)self.layer];

// Create a framebuffer
GLuint framebuffer;
glGenFramebuffers(1, &framebuffer);
glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_STENCIL_ATTACHMENT, GL_RENDERBUFFER, renderbuffer);

// Clear
glClearColor(1, 1, 1, 1);
glClear(GL_COLOR_BUFFER_BIT);

// Read vertex shader source
NSString *vertexShaderSource = [NSString stringWithContentsOfFile:[[NSBundle mainBundle] pathForResource:@"VertexShader" ofType:@"vsh"] encoding:NSUTF8StringEncoding error:nil];
const char *vertexShaderSourceCString = [vertexShaderSource cStringUsingEncoding:NSUTF8StringEncoding];

// Create and compile vertex shader
GLuint _vertexShader = glCreateShader(GL_VERTEX_SHADER);
glShaderSource(_vertexShader, 1, &vertexShaderSourceCString, NULL);
glCompileShader(_vertexShader);

// Read fragment shader source
NSString *fragmentShaderSource = [NSString stringWithContentsOfFile:[[NSBundle mainBundle] pathForResource:@"FragmentShader" ofType:@"fsh"] encoding:NSUTF8StringEncoding error:nil];
const char *fragmentShaderSourceCString = [fragmentShaderSource cStringUsingEncoding:NSUTF8StringEncoding];

// Create and compile fragment shader
GLuint _fragmentShader = glCreateShader(GL_FRAGMENT_SHADER);
glShaderSource(_fragmentShader, 1, &fragmentShaderSourceCString, NULL);
glCompileShader(_fragmentShader);

// Create and link program
GLuint program = glCreateProgram();
glAttachShader(program, _vertexShader);
glAttachShader(program, _fragmentShader);
glLinkProgram(program);

// Use program
glUseProgram(program);

// Define geometry
GLfloat square[] = {
    -0.5, -0.5,
    0.5, -0.5,
    -0.5, 0.5,
    0.5, 0.5};

//Send geometry to vertex shader
const char *aPositionCString = [@"a_position" cStringUsingEncoding:NSUTF8StringEncoding];
GLuint aPosition = glGetAttribLocation(program, aPositionCString);
glVertexAttribPointer(aPosition, 2, GL_FLOAT, GL_FALSE, 0, square);
glEnableVertexAttribArray(aPosition);

// Draw
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

// Present renderbuffer
[context presentRenderbuffer:GL_RENDERBUFFER];
}

So much code and nothing happens... I can draw relentlessly wherever I want without a single stencil test stopping me.

What can I do? How do I check if the stencil buffer has something drawn inside it? If there's a missing puzzle for any of you, I will happily share any other parts of the code.

Any help is greatly appreciated! This has been torturing me for a while now. I will be forever in your debt!

UPDATE

I got the contour thing to work but I didn't use the stencil buffer. I created masks for every drawing area and textures for each mask which I loaded in the fragment shader along with the brush texture. When I tap on an area, I iterate through the array of masks and see which one was selected and bind the mask texture. I will make another post on SO with a more appropriate title and explain it there.

Answer 1

The way you allocate the renderbuffer storage looks problematic:

[context renderbufferStorage:GL_RENDERBUFFER fromDrawable:(CAEAGLLayer*)self.layer];

The documentation says about this method:

The width, height, and internal color buffer format are derived from the characteristics of the drawable object.

The way I understand it, since your "drawable object" will normally be a color buffer, this will create a color renderbuffer. But you need a renderbuffer with stencil format in your case. I'm not sure if there's a way to do this with a utility method in the context class (the documentation says something about "overriding the internal color buffer format"), but the easiest way is probably to simply call the corresponding OpenGL function directly:

glRenderbufferStorage(GL_RENDERBUFFER, GL_STENCIL_INDEX8, width, height);

If you want to use your own FBO for this rendering, you will also need to create a color buffer for it, and attach it to the FBO. Otherwise you're not really producing any rendering output.

Instead of creating a new FBO, it might be easier to make sure that the default framebuffer has a stencil buffer, and render to it directly. To do this, you can request a stencil buffer for your GLKView derived view by making this call during setup:

[view setDrawableStencilFormat: GLKViewDrawableStencilFormat8];