android.grapics.matrix to OpenGL 2.0 ES texture translation

Question

Currently, the application displays an ImageView that successfully zooms and pans around on the screen. On top of that image, I would like to provide a texture that I would like to update itself to zoom or pan when the image below it zooms or pans.

As I understand, this should be possible with using the getImageMatrix() of my current setup on the ImageView and then applying that to my textured bitmap that is on top of the original image.

Edit & Resolution: (with strong aide from the selected answer below)

Currently, the panning in the texture occurs at a different speed than that of the ImageView, but when that has been resolved I will update this posting with additional edits and provide the solution to the entirety of the application. Then only the solution and a quick problem description paragraph will remain. Perhaps I'll even post a surface view and renderer source code for using a zoomable surface view.

In order to accomplish a mapping of an ImageView to an OpenGL texture, there were a couple of things that needed to be done in order to accomplish this correctly. And hopefully, this will aide other people who might want to use a zoomable SurfaceView in the future.

The shader code is written below. The gl_FragColor takes a translation matrix of a 3x3 from the getImageMatrix of any ImageView in order to update the screen from within the onDraw() method.

private final String vertexShader_ =
        "attribute vec4 a_position;\n" +
        "attribute vec4 a_texCoord;\n" +
        "varying vec2 v_texCoord;\n" +
        "void main() {\n" +
        "  gl_Position = a_position;\n" +
        "  v_texCoord = a_texCoord.xy;\n" +
        "}\n";

private final String fragmentShader_ =
        "precision mediump float;\n" +
        "varying vec2 v_texCoord;\n" +
        "uniform sampler2D texture;\n" +
        "uniform mat3 transform;\n" +
        "void main() {\n" +
        "  vec2 uv = (transform * vec3(v_texCoord, 1.0)).xy;\n" +
        "  gl_FragColor = texture2D( texture, uv );\n" +
        "}\n";

At the start, the translation matrix for the OpenGL is simply an identify matrix and our ImageView will update these shared values with the shader through a listener.

private float[] translationMatrix = {1.0f, 0.0f, 0.0f,
                                     0.0f, 1.0f, 0.0f,
                                     0.0f, 0.0f, 1.0f};
...
uniforms_[TRANSLATION_UNIFORM] = glGetUniformLocation(program_, "transform");
checkGlError("glGetUniformLocation transform");
if (uniforms_[TRANSLATION_UNIFORM] == -1) {
    throw new RuntimeException("Could not get uniform location for transform");
}
....
glUniformMatrix3fv(uniforms_[TRANSLATION_UNIFORM], 1, false, FloatBuffer.wrap(translationMatrix));
....

However, the OpenGL code needs an inverse of the OpenGL matrix calculations that come from Android, as well as a transpose operation performed on them before they are handed off to the renderer for being displayed on the screen. This has to do with the way that that information is stored.

protected void onMatrixChanged()
{
    //...
    float[] matrixValues = new float[9];
    Matrix imageViewMatrix = getImageViewMatrix();
    Matrix invertedMatrix = new Matrix();
    imageViewMatrix.invert(invertedMatrix);
    invertedMatrix.getValues(matrixValues);
            transpose(matrixValues);
    matrixChangedListener.onTranslation(matrixValues);
    //...
}

In addition to the fact that the values are in the wrong locations for input into the OpenGL renderer, we also have the problem that we are dealing with our translations on the scale of imageHeight and imageWidth instead of the normalized [0, 1] range that OpenGL expects. So, in order to correct with this we have to target the last column with divisible numbers of our width and height.

matrixValues[6] = matrixValues[6] / getImageWidth();
matrixValues[7] = matrixValues[7] / getImageHeight();

Then, after you have accomplished this, you can start mapping with the ImageView matrix functions or the updated ImageTouchView matrix functions (which have some additional nice features when handling images on Android).

The effect can be seen below with the ImageView behind it taking up the whole screen and the texture in front of it being updated based upon updates from the ImageView. This handles zooming and panning translations between the two.

The only other step in the process would be to change the size of the SurfaceView to match that of the ImageView in order to get the full screen experience when the user is zooming in. In order to get only a surfaceview shown, you can simply put an 'invisible' imageview behind it in the background. And just let the ImageTouchView updates change the way that your OpenGL SurfaceView is shown.

Vote up on the answer below, as they helped me offline through chat significantly to get to this point in understanding what needs to be done for linking an imageview with a texture. They deserve it.

If you do end up going with the vertex shader modification below, instead of the fragment shader resolution, then it would seem that the values no longer need to be inverted to work with the pinch zoom anymore (and in fact work backwards if you leave it as is). Additionally, panning seems to work backwards than is expected as well.

Answer 1

You could forge fragment shader for that, just set this matrix as uniform parameter of the fragment shaderand modify texture coordinates using this matrix.

precision mediump float;
uniform   sampler2D   samp;
uniform   mat3        transform;
varying   vec2        texCoord;

void main()
{
   vec2 uv = (transform * vec3(texCoord, 1.0)).xy;
   gl_FragColor = texture2D(samp, uv);
}

Note that matrix you get from image view is row-major order you should transform it to get OpenGL column-major order, also you would have divide x and y translation components by width and height respectively.

Update: It just occured to me that modifying vertex shader would be better, you can just scale your quad with the same matrix you used in fragment shader, and it would be clamped to your screen if to big, and it will not have a problem with edge clamping when small, and matrix multiplication will happen only 4 times instead of one for each pixel. Just do in vertex shader:

gl_Position = vec4((transform * vec3(a_position.xy, 1.0)).xy, 0.0, 1.0);

and use untransformed texCoord in fragment shader.