Boundary fill incorrectly stops recursion?

Question

I have been trying to execute a simple boundary fill program on my m1 mac using visual studio code where I had the setup, included all the libraries configuring default build task and the build is getting done nicely.

But the issue is when the window is visible, the program has a mouse click event listener, on click it should start region filling with desired color, but it seems to be stopping after drawing one line only.

Here is my program,

#include<stdio.h>
#include <GLUT/glut.h>

int xmin, ymin, xmax, ymax; //Polygon boundaries

float FillColor[3] = {1.0, 0.0, 0.0};  //Color to be filled - red
float BorderColor[3] = {0.0, 0.0, 0.0};  // Border color of polygon - black

void setPixel(int x, int y)
{
    glBegin(GL_POINTS);
        glColor3fv(FillColor);
        glVertex2i(x, y);
    glEnd();
    glFlush();
}
void display()
{
    glClear(GL_COLOR_BUFFER_BIT);

    //Drawing polygon
    glColor3fv(BorderColor);
    glLineWidth(6);
    glBegin(GL_LINES);
        glVertex2i(xmin, ymin);
        glVertex2i(xmin, ymax);
    glEnd();
    glBegin(GL_LINES);
        glVertex2i(xmax, ymin);
        glVertex2i(xmax, ymax);
    glEnd();
    glBegin(GL_LINES);
        glVertex2i(xmin, ymin);
        glVertex2i(xmax, ymin);
    glEnd();
    glBegin(GL_LINES);
        glVertex2i(xmin, ymax);
        glVertex2i(xmax, ymax);
    glEnd();
    glFlush();
}
void BoundaryFill(int x,int y)
{
    float CurrentColor[3];
    glReadPixels(x, y, 1.0, 1.0, GL_RGB, GL_FLOAT, CurrentColor);

    // if CurrentColor != BorderColor and CurrentColor != FillColor
    if((CurrentColor[0] != BorderColor[0] && (CurrentColor[1]) != BorderColor[1] &&
       (CurrentColor[2])!= BorderColor[2]) && (CurrentColor[0] != FillColor[0] &&
       (CurrentColor[1]) != FillColor[1] && (CurrentColor[2]) != FillColor[2]))
    {
        setPixel(x, y);
        BoundaryFill(x+1, y);
        BoundaryFill(x-1, y);
        BoundaryFill(x, y+1);
        BoundaryFill(x, y-1);
        //Using 4-connected approach, remove comment from below lines to make it 8-connected approach
        BoundaryFill(x+1, y+1);
        BoundaryFill(x+1, y-1);
        BoundaryFill(x-1, y+1);
        BoundaryFill(x-1, y-1);
    }
}
void mouse(int btn, int state, int x, int y)
{
    if(btn == GLUT_LEFT_BUTTON && state == GLUT_DOWN)
    {
        printf("%d, %d\n", x, y);
        BoundaryFill(x, 500-y);
    }
}

void init()
{
    glClearColor(0.101, 1.0, 0.980, 1.0); //Background color - cyan
    glMatrixMode(GL_PROJECTION);
    gluOrtho2D(0, 500, 0, 500);
}
int main(int argc, char** argv)
{
    printf("Window size - 500x500 i.e. range of x and y is 0 -> 500\n");
    printf("\nEnter polygon boundaries:-\n");
    printf("Enter xmin: ");
    scanf("%d", &xmin);
    printf("Enter ymin: ");
    scanf("%d", &ymin);
    printf("Enter xmax: ");
    scanf("%d", &xmax);
    printf("Enter ymax: ");
    scanf("%d", &ymax);

    glutInit(&argc, argv);
    glutInitDisplayMode (GLUT_SINGLE | GLUT_RGB);
    glutInitWindowSize(500, 500);
    glutCreateWindow("Boundary-Fill Algorithm");

    init();
    glutDisplayFunc(display);
    glutMouseFunc(mouse);
    glutMainLoop();
    return 0;
}

Please check the image -- where after this stage, the recursion stops which meant to be filling more pixels in 8 connected approach.

Answer 1

The orthographic projection covers an area of [0, 500]^2 pixels, whereas the algorithm assumes an area of [0, 500)^2 pixels. Note the difference between the inclusive range and the exclusive range. This may cause glReadPixels to read the value of the pixel left (and/or down) of where you'd expect it to read, which breaks the algorithm. To solve this, use different dimensions for glOrtho2D:

gluOrtho2D(0, 499, 0, 499);

Additionally, note that the width and height parameters of glReadPixels should be integers, i.e.:

glReadPixels(x, y, 1, 1, GL_RGB, GL_FLOAT, CurrentColor);

and that it is not a good idea to compare float values directly (see elsewhere on StackOverflow). Instead, use something like this:

#include <math.h>

int isEqualFloat(float x, float y)
{
    return fabs(x - y) < 0.001F;
}

I'm assuming the use of glReadPixels is here for educational purposes. A better approach would be to store a boolean matrix for "visited" pixels, which avoids the need for comparing colors in floating point values and retrieving data from GPU memory.

Lastly, using a recent Mac may have influence. For some low resolution program, the OS decides to repeat every pixel to avoid having a very small window. However, this effectively doubles the resolution without OpenGL knowing about it.