Android Camera2 pipeline for preview and MediaCodec input Surface

Question

I'm trying to send Android Camera2 output to both a preview Surface and a surface obtained from MediaCodec.createInputSurface(). However, when I pass those surfaces to a call to CameraDevice.createCaptureSession and then try to build a CaptureRequest, I get:

android.hardware.camera2.CameraAccessException: CAMERA_ERROR (3): submitRequestList - cannot use a surface that wasn't configured.

The CaptureRequest building logic (see below) is from an offical Flutter camera plugin and works fine when you use MediaRecorder.getSurface() instead of MediaCodec.createInputSurface(). Which suggests that the MediaCodec surface hasn't been configured. I'm using a VideoEncoder class from well-tried open source RTMP code (https://github.com/pedroSG94/rtmp-rtsp-stream-client-java) that works with the old Camera API (i.e. not Camera2). That class initialises the codec thus:

   String type = CodecUtil.H264_MIME;
   MediaCodecInfo encoder = chooseEncoder(type);
    try {
      if (encoder != null) {
        codec = MediaCodec.createByCodecName(encoder.getName());
      } else {
        Log.e(TAG, "Valid encoder not found");
        return false;
      }
      MediaFormat videoFormat;
      //if you dont use mediacodec rotation you need swap width and height in rotation 90 or 270
      // for correct encoding resolution
      String resolution;
      if (!hardwareRotation && (rotation == 90 || rotation == 270)) {
        resolution = height + "x" + width;
        videoFormat = MediaFormat.createVideoFormat(type, height, width);
      } else {
        resolution = width + "x" + height;
        videoFormat = MediaFormat.createVideoFormat(type, width, height);
      }
      Log.i(TAG, "Prepare video info: " + this.formatVideoEncoder.name() + ", " + resolution);
      videoFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT,
          this.formatVideoEncoder.getFormatCodec());
      videoFormat.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 0);
      videoFormat.setInteger(MediaFormat.KEY_BIT_RATE, bitRate);
      videoFormat.setInteger(MediaFormat.KEY_FRAME_RATE, fps);
      videoFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, iFrameInterval);
      if (hardwareRotation) {
        videoFormat.setInteger("rotation-degrees", rotation);
      }
      if (this.avcProfile > 0 && this.avcProfileLevel > 0) {
        // MediaFormat.KEY_PROFILE, API > 21
        videoFormat.setInteger("profile", this.avcProfile);
        // MediaFormat.KEY_LEVEL, API > 23
        videoFormat.setInteger("level", this.avcProfileLevel);
      }
      codec.configure(videoFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
      inputSurface = codec.createInputSurface();

The code that fails when I try to build a capture request is in CameraCaptureSession.StateCallback.onConfigured, where the call to build() raises the exception:

createCaptureSession( CameraDevice.TEMPLATE_RECORD, successCallback, surfaceFromMediaCodec );

private void createCaptureSession(
        int templateType, Runnable onSuccessCallback, Surface... surfaces)
        throws CameraAccessException {

    // Close any existing capture session.
    closeCaptureSession();

    // Create a new capture builder.
    captureRequestBuilder = cameraDevice.createCaptureRequest(templateType);

    // Build Flutter surface to render to
    SurfaceTexture surfaceTexture = flutterTexture.surfaceTexture();
    surfaceTexture.setDefaultBufferSize(previewSize.getWidth(), previewSize.getHeight());
    Surface flutterSurface = new Surface(surfaceTexture);
    captureRequestBuilder.addTarget(flutterSurface);

    List<Surface> remainingSurfaces = Arrays.asList(surfaces);
    if (templateType != CameraDevice.TEMPLATE_PREVIEW) {
        // If it is not preview mode, add all surfaces as targets.
        for (Surface surface : remainingSurfaces) {
            captureRequestBuilder.addTarget(surface);
        }
    }

    // Prepare the callback
    CameraCaptureSession.StateCallback callback =
            new CameraCaptureSession.StateCallback() {
                @Override
                public void onConfigured(@NonNull CameraCaptureSession session) {
                    try {
                        if (cameraDevice == null) {
                            dartMessenger.send(
                                    DartMessenger.EventType.ERROR, "The camera was closed during configuration.");
                            return;
                        }
                        cameraCaptureSession = session;
                        captureRequestBuilder.set(
                                CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_MODE_AUTO);
                        cameraCaptureSession.setRepeatingRequest(captureRequestBuilder.build(), null, null);
                        if (onSuccessCallback != null) {
                            onSuccessCallback.run();
                        }
                    } catch (CameraAccessException | IllegalStateException | IllegalArgumentException e) {
                        Log.i( TAG, "exception building capture session " + e );
                        dartMessenger.send(DartMessenger.EventType.ERROR, e.getMessage());
                    }
                }

                @Override
                public void onConfigureFailed(@NonNull CameraCaptureSession cameraCaptureSession) {
                    dartMessenger.send(
                            DartMessenger.EventType.ERROR, "Failed to configure camera session.");
                }
            };

    // Collect all surfaces we want to render to.
    List<Surface> surfaceList = new ArrayList<>();
    surfaceList.add(flutterSurface);
    surfaceList.addAll(remainingSurfaces);
    // Start the session
    cameraDevice.createCaptureSession(surfaceList, callback, null);
}

If I remove the MediaCodec inputSurface as a build target, it works (but I don't capture anything into the MediaCodec). What am I missing? BTW, there are bits of Flutter code in the second code extract but there's no evidence that the embedding in Flutter is relevant.

Answer 1

Answering my own question. I was thrown off the scent by the misleading Exception message "cannot use a surface that wasn't configured". The surface was configured. And I thought I'd checked the sizes, but one was 720x480, the other was 480x720. It worked after I swapped.