Capture camera preview greater than screen size in Android

Question

I have a real-time video processing app in Android that captures from camera and display the preview to a TextureView.

The issue is that I want to capture the camera preview at 2k, but screen size is 1080, so TextureView size is max of 1080.

I want to be able to get, TextureView.getBitmap() and get a 2k image, but because TextureView is maxed to the screen size the image is only 1080px.

I found that I could overwrite the TextureView onMeasure() method to force the TextureView size to 2k. This works, however because the screen is 1080, the phone only shows the middle of the 2k image, how to display the whole image?

My 2nd attempt I made the layout size of the TextureView 0 so it was hidden, and added another ImageView to the layout, then from the listener onImageAvailable() I convert the Image to a Bitmap and set it in the ImageView. This works, but is very slow, get a delayed image instead of live camera preview.

I think I need to be able to zoom out the TextureView, or perhaps have 2 texture views and copy from one to the other. But can't get it to work as desired.

Please, any help or suggestions.

protected void onMeasure(final int widthMeasureSpec, final int heightMeasureSpec) { super.onMeasure(widthMeasureSpec, heightMeasureSpec); width = CameraConnectionFragment.MINIMUM_PREVIEW_SIZE; height = CameraConnectionFragment.MINIMUM_PREVIEW_SIZE; setMeasuredDimension(width, height); }

The code is similar to, https://github.com/miyosuda/TensorFlowAndroidDemo/tree/master/app/src/main/java/org/tensorflow/demo

The camera code is,

private void openCamera(final int width, final int height) {
        setUpCameraOutputs(width, height);
        configureTransform(width, height);
        final Activity activity = getActivity();
        final CameraManager manager = (CameraManager) activity.getSystemService(Context.CAMERA_SERVICE);
        manager.openCamera(cameraId, stateCallback, backgroundHandler);
    }

Camera preview code is,

private void createCameraPreviewSession() {
        try {
            final SurfaceTexture texture = textureView.getSurfaceTexture();
            assert texture != null;

            // We configure the size of default buffer to be the size of camera preview we want.
            texture.setDefaultBufferSize(previewSize.getWidth(), previewSize.getHeight());

            // This is the output Surface we need to start preview.
            final Surface surface = new Surface(texture);

            // We set up a CaptureRequest.Builder with the output Surface.
            previewRequestBuilder = cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
            previewRequestBuilder.addTarget(surface);

            // Create the reader for the preview frames.
            previewReader =
                ImageReader.newInstance(
                    previewSize.getWidth(), previewSize.getHeight(), ImageFormat.YUV_420_888, 2);

            previewReader.setOnImageAvailableListener(imageListener, backgroundHandler);
            previewRequestBuilder.addTarget(previewReader.getSurface());
            zoom(zoom, activity, false);
            // Here, we create a CameraCaptureSession for camera preview.
            cameraDevice.createCaptureSession(
                Arrays.asList(surface, previewReader.getSurface()),
                new CameraCaptureSession.StateCallback() {
                    @Override
                    public void onConfigured(final CameraCaptureSession cameraCaptureSession) {
                        // The camera is already closed
                        if (null == cameraDevice) {
                            return;
                        }

                        // When the session is ready, we start displaying the preview.
                        captureSession = cameraCaptureSession;
                        try {
                            // Auto focus should be continuous for camera preview.
                            previewRequestBuilder.set(
                                CaptureRequest.CONTROL_AF_MODE,
                                CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE);
                            // Flash is automatically enabled when necessary.

                            previewRequestBuilder.set(CaptureRequest.CONTROL_AE_MODE, CameraMetadata.CONTROL_AE_MODE_ON);
                            if (isTorchOn) {
                                previewRequestBuilder.set(CaptureRequest.FLASH_MODE, CameraMetadata.FLASH_MODE_TORCH);
                            } else {
                                previewRequestBuilder.set(CaptureRequest.FLASH_MODE, CameraMetadata.FLASH_MODE_OFF);
                            }
                            // Finally, we start displaying the camera preview.
                            previewRequest = previewRequestBuilder.build();
                            captureSession.setRepeatingRequest(previewRequest, captureCallback, backgroundHandler);
                        } catch (final CameraAccessException exception) {
                            LOGGER.e(exception, "Exception!");
                        } catch (Exception exception) {
                            Log.wtf("Camera preview", exception);
                        }
                    }
                },
                null);
        } catch (final Exception e) {
            LOGGER.e(e, "Exception!");
        }

Answer 1

One possible solution i can think is to extend your custom TextureView and override the onMeasure() method as you suggested in your question to force the TextureView size to be 2k like:

@Override
protected void onMeasure(final int widthMeasureSpec, final int heightMeasureSpec) {
    super.onMeasure(widthMeasureSpec, heightMeasureSpec);
    setMeasuredDimension(2000, 2000);
}

and make the TextureView scrollable in both directions (vertically and horizontally) something like:

<?xml version="1.0" encoding="utf-8"?>
<HorizontalScrollView xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content">

    <LinearLayout
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:orientation="horizontal">

        <ScrollView
            android:layout_width="match_parent"
            android:layout_height="wrap_content"
            android:fillViewport="true">

            <LinearLayout
                android:layout_width="match_parent"
                android:layout_height="wrap_content"
                android:orientation="vertical">

                <mypackage.name.MyTextureView
                    android:id="@+id/textureView"
                    android:layout_width="wrap_content"
                    android:layout_height="wrap_content" />

            </LinearLayout>
        </ScrollView>
    </LinearLayout>
</HorizontalScrollView>

And later retrieve the bitmap with textureView.getBitmap(); in 2k resolution.

Answer 2

You are doing weird things. Generally, you have no control over the max and min preview size. Method

CameraCharacteristics = cameraManager.getCameraCharacteristics(cameraId)
val supportedSizes: Size[] = cameraCharacteristics.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP)
                              .getOutputSizes(SurfaceTexture::class.java)

will return the complete list of all the sized you can use. It is an implementation limitation - you cannot overcome it for a specific textureView implementation of a specific device. All you can do is choose the correct aspect ratio.

Capture sizes will be different and usually much bigger and the list will be bigger also - the issue is that you won't be able to show that in the preview - the captured data will be visible only in the output file/stream.

Although! There are several approaches you can use here in order to achieve what you want.

1. Modify the preview stream as you want - interpolate(there is plenty of info about that online) to make it larger both size- and quality-wise or just upscale losing quality. (it will still be displayed on the TextureView that is limited by its implementation)
2. Implement your own version of texture view that does all this magic with OpenGL inside the TextureView - it will be much more performant but relatively harder to implement.
3. Try to use the output capture stream(or even preview one) as a source for some TextureView that already knows how to show quality pictures - the one from some video player.
4. Maybe something else.

Personally, I implemented the first two approaches a long time ago and I can say that they are possible and relatively easy to implement. I won't provide code or techniques because it is quite a big amount of code and I don't remember all the methods well enough.

The third approach is also possible and I have seen it in work but never implemented myself(only fixed bugs) (in my case it was implemented on the native side - so maybe you will need C for that.)

I know the answer may not be as direct as you wanted but the task is not that trivial also. Either way, hope it helps.