How do I render smooth fonts (any "flat" shape really) with OpenGL?

Question

I created the ftmesh library to draw fonts on my OpenGL ES 3 screen. The code works, but the edges of the characters is raggedy. I'm using an NVidia Jetson AGX Xavier with OpenGL ES 3 and the glShadeModel(GL_SMOOTH) doesn't have any effect (and glEnable(GL_POLYGON_SMOOTH) says Invalid Enum—since it's not allowed in ES 2 or 3).

Here is an example of a screenshot showing the ugly edges:

Is the only option to add an offset edge all around and make the outside part 100% transparent?

Or is there a way to do it with a shader? It would be fantastic to get a link to an example.

My current shader just transmits the vertices as is:

varying vec4 polygon_color;

attribute vec2 in_position;
attribute vec4 in_color;

void main()
{
    polygon_color = in_color;
    gl_Position = vec4(in_position.xy, 0, 1);
}

And here is the fragment shader:

varying vec4 polygon_color;

void main()
{
    gl_FragColor = polygon_color;
}

I set all the in_color vertices to (1.0f, 1.0f, 1.0f, 0.75f) before the glDrawArrays() call.

Sample Buffers (Solution)

As suggested by Rabbid76 in a comment, I tried adding multisampling. It has an effect, it created a form of grid, though... much more transparent than expected (see picture below).

1. In my g_egl_visual_attributes I added EGL_SAMPLE_BUFFERS like so:

    constexpr EGLint const g_egl_visual_attributes[] =
    {
        EGL_RED_SIZE, 8,
        EGL_GREEN_SIZE, 8,
        EGL_BLUE_SIZE, 8,
        EGL_ALPHA_SIZE, 8,
        EGL_SURFACE_TYPE, EGL_WINDOW_BIT,
        EGL_SAMPLE_BUFFERS, 1,   // <-- added this option
        EGL_SAMPLES, 8,          //     and this one
        EGL_NONE,
    };
   

   Note: with my configuration, EGL_SAMPLES of 16 doesn't work (no such buffer available).

2. Then I enabled the sample alpha coverage

    glEnable(GL_SAMPLE_ALPHA_TO_COVERAGE);
    glEnable(GL_SAMPLE_COVERAGE);
   

   But that had the side effect shown below...

   

3. So I tried without the glEnable() calls and now that works as expected]!

   If you look closely, in this last image we can see the anti-aliasing on the edges:

   

   So MSAA resolved the issue.

Answer 1

A much nicer anti-aliasing is provided by the library FreeTypeGL, but it renders the characters in a completely different manner, by means of a distance map, and as a side effect it doesn't make perfect corners:

https://github.com/rougier/freetype-gl