OpenGL video frame fitting

Question

I am working on a project where I have to project the data from a camera with a resolution of 640x480 on a 4K screen in portrait mode.

The camera is the Kinect V1 but I will switch to version 2 with a better resolution (1920x1080).

My question is how to change the scale of a texture to display in order to get a correct result. For the moment, I have managed to display on the entire screen but the image is flattened in width. The ideal would be to keep the proportionality and cut an X width on each side of the image.

I am using SDL with OpenGL, here is the concerned part of the code:

// window creation
auto window = SDL_CreateWindow("Imagine",
                                   x,
                                   y,
                                   0,
                                   0,
                                   SDL_WINDOW_OPENGL | SDL_WINDOW_SHOWN | SDL_WINDOW_FULLSCREEN_DESKTOP | SDL_WINDOW_ALLOW_HIGHDPI | SDL_WINDOW_BORDERLESS);

// GL initialization and texture creation
void SdlNuitrackRenderHandler::initTexture(int width, int height)
{
    glEnable(GL_TEXTURE_2D);
    glEnableClientState(GL_VERTEX_ARRAY);
    glDisableClientState(GL_COLOR_ARRAY);
    glOrtho(0, _width, _height, 0, -1.0, 1.0);
    glMatrixMode(GL_PROJECTION);
    glPushMatrix();
    glLoadIdentity();

    glGenTextures(1, &_textureID);

    width = power2(width);
    height = power2(height);

    if (_textureBuffer != 0)
        delete[] _textureBuffer;

    _textureBuffer = new uint8_t[width * height * 3];
    memset(_textureBuffer, 0, sizeof(uint8_t) * width * height * 3);

    glBindTexture(GL_TEXTURE_2D, _textureID);
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, width, height, 0, GL_RGB, GL_UNSIGNED_BYTE, NULL);

    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

    // Set texture coordinates [0, 1] and vertexes position
    {
        _textureCoords[0] = (float) _width / width;
        _textureCoords[1] = (float) _height / height;
        _textureCoords[2] = (float) _width / width;
        _textureCoords[3] = 0.0;
        _textureCoords[4] = 0.0;
        _textureCoords[5] = 0.0;
        _textureCoords[6] = 0.0;
        _textureCoords[7] = (float) _height / height;

        _vertexes[0] = _width;
        _vertexes[1] = _height;
        _vertexes[2] = _width;
        _vertexes[3] = 0.0;
        _vertexes[4] = 0.0;
        _vertexes[5] = 0.0;
        _vertexes[6] = 0.0;
        _vertexes[7] = _height;
    }

// Texture rendering
// Render prepared background texture
void SdlNuitrackRenderHandler::renderTexture()
{
    glClearColor(1, 1, 1, 1);
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

    glEnable(GL_TEXTURE_2D);
    glColor4f(1, 1, 1, 1);

    glBindTexture(GL_TEXTURE_2D, _textureID);
    glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, _width, _height, GL_RGB, GL_UNSIGNED_BYTE, _textureBuffer);

    glEnableClientState(GL_VERTEX_ARRAY);
    glEnableClientState(GL_TEXTURE_COORD_ARRAY);

    glVertexPointer(2, GL_FLOAT, 0, _vertexes);
    glTexCoordPointer(2, GL_FLOAT, 0, _textureCoords);

    glDrawArrays(GL_TRIANGLE_FAN, 0, 4);

    glDisableClientState(GL_VERTEX_ARRAY);
    glDisableClientState(GL_TEXTURE_COORD_ARRAY);

    glDisable(GL_TEXTURE_2D);
}

Answer 1

While I agree with what was written in the comments about this being out-dated OpenGL code, the issue has nothing to do with OpenGL at its heart. You want to draw 1 rectangle with the correct aspect ratio inside another rectangle that has a different aspect ratio. You simply need to know where to place the vertices.

Typically with the TEXTURE_2D texture target, you want your texture coordinates to be 0-1 in both directions, unless you plan to crop the input image. There was a time when textures had to have a width and height that were a power of 2. That hasn't been the case in a very long time. So remove these 2 lines:

width = power2(width);
height = power2(height);

So the first thing is to set those properly:

    _textureCoords[0] = 1.0;
    _textureCoords[1] = 1.0;
    _textureCoords[2] = 1.0;
    _textureCoords[3] = 0.0;
    _textureCoords[4] = 0.0;
    _textureCoords[5] = 0.0;
    _textureCoords[6] = 0.0;
    _textureCoords[7] = 1.0;

(Consequently, that code is really hard to read and will be a pain to maintain. You should make the texture coordinates (and vertex coordinates) be a struct with an x and y value so it makes sense. Right now it's not obvious that it's 4 sets of 2D coordinates that are (max, max), (max, min), (min, min), (min, max). But I digress.)

Next, to figure out the texture coordinates, you need to know whether the video is going to be scaled to fit the width or the height. To do this, you can figure out

double widthScaleRatio = displayWidth / imageWidth; // <- using this scale will guarantee the width of the new image is the same as the display's width, but might crop the height
double heightScaleRatio = displayHeight / imageHeight; // <- using this scale will guarantee the height of the new image is the same as the display's height but might crop the width
double  scale = 1.0;
// If scaling by the widthScaleRatio makes the height too big, use the heightScaleRatio
// Otherwise use the widthScaleRatio
if (imageHeight * widthScaleRatio > displayHeight)
{
    scale = heightScaleRatio;
}
else
{
    scale = widthScaleRatio;
}

Now scale you width and height by the scale:

double newWidth  = imageWidth * scale;
double newHeight = imageHeight * scale;

and set your vertices based on that:

    _vertexes[0] = newWidth;
    _vertexes[1] = newHeight;
    _vertexes[2] = newWidth;
    _vertexes[3] = 0.0;
    _vertexes[4] = 0.0;
    _vertexes[5] = 0.0;
    _vertexes[6] = 0.0;
    _vertexes[7] = newHeight;

And the same caveat applies to making this code easier to read as with the texture coordinates.

EDIT: Here's a simple program to show how it works:

int main(){
    double  displayWidth    = 2160;
    double  displayHeight   = 4096;
    double  imageWidth  =  640;
    double  imageHeight = 480;
    double widthScaleRatio = displayWidth / imageWidth; // <- using this scale will guarantee the width of the new image is the same as the display's width, but might crop the height
    double heightScaleRatio = displayHeight / imageHeight; // <- using this scale will guarantee the height of the new image is the same as the display's height but might crop the width
    double  scale = 1.0;
    // If scaling by the widthScaleRatio makes the height too big, use the heightScaleRatio
    // Otherwise use the widthScaleRatio
    if (imageHeight * widthScaleRatio > displayHeight)
    {
        scale = heightScaleRatio;
    }
    else
    {
        scale = widthScaleRatio;
    }

    double newWidth  = imageWidth * scale;
    double newHeight = imageHeight * scale;

    std::cout << "New size = (" << newWidth << ", " << newHeight  << ")\n";
}

When I run it, I get:

New size = (2160, 1620)