How to rotate a iPhone camera output OpenGLES texture

Question

I'm having trouble mapping the iPhone camera output on a simple plane in OpenGLES 2.0. For some reason the OpenGL texture generated through the CVOpenGLESTextureCacheCreateTextureFromImage is flipped and rotated.

Here's my situation: I made a Wavefront .obj loader that parses this simple plane object:

v -0.5 -0.5 0
v 0.5 -0.5 0
v -0.5 0.5 0
v 0.5 0.5 0

vt 0 0 0
vt 1 0 0
vt 0 1 0
vt 1 1 0

f 4/4 3/3 1/1 
f 2/2 4/4 1/1 

As far as I'm concerned my texture mapping works like it has to. I've verified this by mapping a simple png image onto the plane. It shows up correctly.

(Just for the record: I loaded the image in the iPhone coordinate system, flipped the image pixels to match the OpenGL's top-left texture coordinate system and mapped everything). Result: works!

So, after shuffling my code around a bit and feeding the camera pixel buffer into my texture unit something goes wrong (or maybe this is intended behavior and I should counter that in one way or another? Any suggestions are welcome, as long as it isn't CPU heavy code).

Here's my code: some of it is inspired by Brad Larson's ColorTracking example and Apple's GLCameraRipple example.

Code to capture the output:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    [self.delegate processNewCameraFrame:pixelBuffer];
}

After passing around the image buffer ref it ends up here:

- (void)updateCameraFrame:(CVImageBufferRef)pCameraFrame
{
    if(cameraTextureMaterial)
    {
        cameraTextureMaterial->updateWithCameraFrame(pCameraFrame);
    }
}

CamerTextureMaterial:

void CameraTextureMaterial::updateWithCameraFrame(CVImageBufferRef pixelBuffer)
{
    if(!_videoTextureCache)
    {
        cout << "No video texture cache" << endl;
        return;
    }

    cleanup();

    CVReturn err;
    size_t width = CVPixelBufferGetWidth(pixelBuffer);
    size_t height = CVPixelBufferGetHeight(pixelBuffer);

    glActiveTexture(GL_TEXTURE0);

    err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                       _videoTextureCache,
                                                       pixelBuffer,
                                                       NULL,
                                                       GL_TEXTURE_2D,
                                                       GL_RGBA,
                                                       width,
                                                       height,
                                                       GL_BGRA,
                                                       GL_UNSIGNED_BYTE,
                                                       0,
                                                       &_lumaTexture);
    if (err)
    {
        cout << "Error at CVOpenGLESTextureCacheCreateTextureFromImage" << endl;
    }

    glBindTexture(CVOpenGLESTextureGetTarget(_lumaTexture), CVOpenGLESTextureGetName(_lumaTexture));
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
}

I'm excluding initialization and cleanup of the texture cache since this doesn't matter for my texture orientation problem.

Any suggestions on how to get the correct orientation for my texture?

Answer 1

Solving this issues was actually pretty easy by choosing your texture coordinates wisely. I rotated, flipped and cropped the camera texture by mapping it with the coordinates below and rendered the orthogonal scene to a new texture which I then use in my scene.

Coordinates I used:

void CameraClipPlaneObject3D::init()
{
    // Create plane object
    vertices = shared_ptr<Vertex3DVector>(new Vertex3DVector());
    vertices->push_back(Vertex3D(1, -1, 0));
    vertices->push_back(Vertex3D(1, 1, 0));
    vertices->push_back(Vertex3D(-1, 1, 0));
    vertices->push_back(Vertex3D(-1, -1, 0));

    // Create texture coords
    textureCoords = shared_ptr<Texture2DVector>(new Texture2DVector());
    textureCoords->push_back(Texture2D(0.875, 0));
    textureCoords->push_back(Texture2D(0.125, 0));
    textureCoords->push_back(Texture2D(0.125, 1));
    textureCoords->push_back(Texture2D(0.875, 1));

    // Create indices
    indices = shared_ptr<IndexVector>(new IndexVector());
    indices->push_back(0);
    indices->push_back(1);
    indices->push_back(2);
    indices->push_back(2);
    indices->push_back(3);
    indices->push_back(0);
}

Here's how I render the plane to a new texture:

1. first create a texture like you normally do (don't forget texture filtering)
2. then create a framebuffer like you normally do
3. attach the texture colorbuffer to the color attachment of this new framebuffer:
4. In your rendering loop, bind to this new framebuffer and render your scene.

Some sample code:

void TextureRenderTarget::init()
{
    // Create texture to render to
    texture = new RenderTexture(width, height, GL_LINEAR, GL_LINEAR);

    // Create framebuffer and attach texture to framebuffer
    framebuffer = new Framebuffer(width, height);
    framebuffer->attachTexture(texture);
}

Attach code:

void Framebuffer::attachTexture(Texture *texture) const
{
    glBindFramebuffer(GL_FRAMEBUFFER, handle);
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, texture->getHandle(), 0);
}

Answer 2

I honestly don't see the benefit of using an OBJ for such a simple geometry. In the end we are just talking about 2 triangles :) which are quite easy to be created on the fly in your code.

Apart this "cosmetic" comment, OpenGL works by default on non-flipped images for texturing. Most probably the texture you link from the camera is by default flipped (this depends a lot on the internet data organization of the format, not on what you see when you open the picture).

One quick and dirty solution could be to use the below trick:

glMatrixMode(GL_TEXTURE);
glLoadIdentity();
glScalef(1.0f, -1.0f, 1.0f);
glMatrixMode(GL_MODELVIEW);

This should do the trick for you and save tons of computing power.

Cheers Maurizio