Anti-aliasing/smoothing/supersampling 2d images with opengl

Question

I'm playing with 2D graphics in OpenGL - fractals and other fun stuff ^_^. My basic setup is rendering a couple triangles to fill the screen and using a fragment shader to draw cool stuff on them. I'd like to smooth things out a bit, so I started looking into supersampling. It's not obvious to me how to go about this. Here's what I've tried so far...

First, I looked at the Apple docs on anti-aliasing. I updated my pixel format initialization:

NSOpenGLPixelFormatAttribute attrs[] =
{
    NSOpenGLPFADoubleBuffer,
    NSOpenGLPFADepthSize, 24,
    NSOpenGLPFAOpenGLProfile, NSOpenGLProfileVersion4_1Core,
    NSOpenGLPFASupersample,
    NSOpenGLPFASampleBuffers, 1,
    NSOpenGLPFASamples, 4,
    0
};

I also added the glEnable(GL_MULTISAMPLE); line. GL_MULTISAMPLE_FILTER_HINT_NV doesn't seem to be defined (docs appear to be out of date), so I wasn't sure what to do there.

That made my renders slower but doesn't seem to be doing anti-aliasing, so I tried the "Render-to-FBO" approach described on the OpenGL Wiki on Multisampling. I've tried a bunch of variations, with a variety of outcomes: successful rendering (which don't appear to be anti-aliased), rendering garbage to the screen (fun!), crashes (app evaporates and I get a system dialog about graphics issues), and making my laptop unresponsive aside from the cursor (got the system dialog about graphics issues after hard reboot).

I am checking my framebuffer's status before drawing, so I know that's not the issue. And I'm sure I'm rendering with hardware, not software - saw that suggestion on other posts.

I've spent a fair amount of time on it and still don't quite understand how to approach this. One thing I'd love some help on is how to query GL to see if supersampling is enabled properly, or how to tell how many times my fragment shader is called, etc. I'm also a bit confused about where some of the calls go - most examples I find just say which methods to call, but don't specify which ones need to go in the draw callback. Anybody have a simple example of SSAA with OpenGL 3 or 4 and OSX... or other things to try?

Edit: drawing code - super broken (don't judge me), but for reference:

- (void)draw
{
    glBindVertexArray(_vao);    // todo: is this necessary? (also in init)
    glBufferData(GL_ARRAY_BUFFER, 12 * sizeof(GLfloat), points, GL_STATIC_DRAW);

    glGenTextures( 1, &_tex );
    glBindTexture( GL_TEXTURE_2D_MULTISAMPLE, _tex );
    glTexImage2DMultisample( GL_TEXTURE_2D_MULTISAMPLE, 4, GL_RGBA8, _width * 2, _height * 2, false );

    glGenFramebuffers( 1, &_fbo );
    glBindFramebuffer( GL_FRAMEBUFFER, _fbo );
    glFramebufferTexture2D( GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D_MULTISAMPLE, _tex, 0 );

    GLint status;
    status = glCheckFramebufferStatus( GL_FRAMEBUFFER );
    if (status != GL_FRAMEBUFFER_COMPLETE) {
        NSLog(@"incomplete buffer 0x%x", status);
        return;
    }

    glBindFramebuffer(GL_DRAW_FRAMEBUFFER, 0);
    glBindFramebuffer(GL_READ_FRAMEBUFFER, _fbo);
    glDrawBuffer(GL_BACK);
    glClear(GL_COLOR_BUFFER_BIT);
    glDrawArrays(GL_TRIANGLES, 0, 6);
    glBlitFramebuffer(0, 0, _width * 2, _height * 2, 0, 0, _width, _height, GL_COLOR_BUFFER_BIT, GL_LINEAR);

    glDeleteTextures(1, &_tex);
    glDeleteFramebuffers(1, &_fbo);
    glBindFramebuffer( GL_FRAMEBUFFER, 0 );
}

Update: I changed my code per Reto's suggestion below:

glBindFramebuffer(GL_DRAW_FRAMEBUFFER, _fbo);
glClear(GL_COLOR_BUFFER_BIT);
glDrawArrays(GL_TRIANGLES, 0, 6);
glBindFramebuffer(GL_READ_FRAMEBUFFER, _fbo);
glBindFramebuffer(GL_DRAW_FRAMEBUFFER, 0);
glBlitFramebuffer(0, 0, _width * 2, _height * 2, 0, 0, _width, _height,
                  GL_COLOR_BUFFER_BIT, GL_LINEAR);

This caused the program to render garbage to the screen. I then got rid of the * 2 multiplier, and it still drew garbage to the screen. I then turned off the NSOpenGLPFA options related to multi/super-sampling, and it rendered normally, with no anti-aliasing.

I also tried using a non-multisample texture, but kept getting incomplete attachment errors. I'm not sure if this is due to the NVidia issue mentioned on the OpenGL wiki (will post ina comment since I don't have enough rep to post more than 2 links) or something else. If someone could suggest a way to find out why the attachment is incomplete, that would be very, very helpful.

Finally, I tried using a renderbuffer instead of a texture, and found that specifying width and height greater than the viewport size in glRenderbufferStorage doesn't seem to work as expected.

GLuint rb;
glGenRenderbuffers(1, &rb);
glBindRenderbuffer(GL_RENDERBUFFER, rb);
glRenderbufferStorage(GL_RENDERBUFFER, GL_RGBA8, _width * 2, _height * 2);

// ...

glBindFramebuffer(GL_DRAW_FRAMEBUFFER, _fbo);
glClear(GL_COLOR_BUFFER_BIT);
glDrawArrays(GL_TRIANGLES, 0, 6);
glBindFramebuffer(GL_READ_FRAMEBUFFER, _fbo);
glBindFramebuffer(GL_DRAW_FRAMEBUFFER, 0);
glBlitFramebuffer(0, 0, _width * 2, _height * 2, 0, 0, _width, _height,
                  GL_COLOR_BUFFER_BIT, GL_LINEAR);

... renders in the bottom left hand 1/4 of the screen. It doesn't appear to be smoother though...

Update 2: doubled the viewport size, it's no smoother. Turning NSOpenGLPFASupersample still causes it to draw garbage to the screen. >.<

Update 3: I'm an idiot, it's totally smoother. It just doesn't look good because I'm using an ugly color scheme. And I have to double all my coordinates because the viewport is 2x. Oh well. I'd still love some help understanding why NSOpenGLPFASupersample is causing such crazy behavior...

Answer 1

Your sequence of calls here looks like it wouldn't do what you intended:

glBindFramebuffer(GL_DRAW_FRAMEBUFFER, 0);
glBindFramebuffer(GL_READ_FRAMEBUFFER, _fbo);
glDrawBuffer(GL_BACK);
glClear(GL_COLOR_BUFFER_BIT);
glDrawArrays(GL_TRIANGLES, 0, 6);
glBlitFramebuffer(0, 0, _width * 2, _height * 2, 0, 0, _width, _height, GL_COLOR_BUFFER_BIT, GL_LINEAR);

When you call glClear() and glDrawArrays(), your current draw framebuffer, which is determined by the last call to glBindFramebuffer(GL_DRAW_FRAMEBUFFER, ...), is the default framebuffer. So you never render to the FBO. Let me annotate the above:

// Set draw framebuffer to default (0) framebuffer. This is where the rendering will go.
glBindFramebuffer(GL_DRAW_FRAMEBUFFER, 0);
// Set read framebuffer to the FBO.
glBindFramebuffer(GL_READ_FRAMEBUFFER, _fbo);
// This is redundant, GL_BACK is the default draw buffer for the default framebuffer.
glDrawBuffer(GL_BACK);
// Clear the current draw framebuffer, which is the default framebuffer.
glClear(GL_COLOR_BUFFER_BIT);
// Draw to the current draw framebuffer, which is the default framebuffer.
glDrawArrays(GL_TRIANGLES, 0, 6);
// Copy from read framebuffer (which is the FBO) to the draw framebuffer (which is the
// default framebuffer). Since no rendering was done to the FBO, this will copy garbage
// into the default framebuffer, wiping out what was previously rendered.
glBlitFramebuffer(0, 0, _width * 2, _height * 2, 0, 0, _width, _height,
                  GL_COLOR_BUFFER_BIT, GL_LINEAR);

To get this working, you need to set the draw framebuffer to the FBO while rendering, and then set the read framebuffer to the FBO and the draw framebuffer to the default for the copy:

glBindFramebuffer(GL_DRAW_FRAMEBUFFER, _fbo);
glClear(GL_COLOR_BUFFER_BIT);
glDrawArrays(GL_TRIANGLES, 0, 6);
glBindFramebuffer(GL_READ_FRAMEBUFFER, _fbo);
glBindFramebuffer(GL_DRAW_FRAMEBUFFER, 0);
glBlitFramebuffer(0, 0, _width * 2, _height * 2, 0, 0, _width, _height,
                  GL_COLOR_BUFFER_BIT, GL_LINEAR);

Recap:

• Draw commands write to GL_DRAW_FRAMEBUFFER.
• glBlitFramebuffer() copies from GL_READ_FRAMEBUFFER to GL_DRAW_FRAMEBUFFER.

A couple more remarks on the code:

• Since you're creating a multisample texture of twice the size, you're using both multisampling and supersampling at the same time:

  glBindTexture(GL_TEXTURE_2D_MULTISAMPLE, _tex);
  glTexImage2DMultisample(GL_TEXTURE_2D_MULTISAMPLE, 4, GL_RGBA8,
                          _width * 2, _height * 2, false);
  

  Which is entirely legal. But it's... a lot of sampling. If you just want supersampling, you can use a regular texture.

• You could use a renderbuffer instead of a texture for the FBO color target. No huge advantage, but it's simpler, and potentially more efficient. You only need to use textures as attachments if you want to sample the result later, which is not the case here.