CODEWITHSUNDEEP
androidvideoandroid-ndkandroid-mediacodec
Mohit Bhola
Mohit Bhola

Reputation: 41

Getting MediaCodec decoder provide video frames in RGBA

I am trying to use a MediaCodec decoder (via the NDK API) to fetch video frames (for further processing) from a .mp4 file. Here is the sample code that sets up the decoder to render to a surface (owned by an ImageReader):

// Omitting most error handling for clarity

AMediaExtractor* ex = AMediaExtractor_new();
media_status_t err = AMediaExtractor_setDataSourceFd(ex, fd /*opened previously*/, outStart, outLen);
close(fd);

int numtracks = AMediaExtractor_getTrackCount(ex);
AMediaCodec* decoder = NULL;

for (int i = 0; i < numtracks; i++) {
    AMediaFormat *format = AMediaExtractor_getTrackFormat(ex, i);
    const char *s = AMediaFormat_toString(format);
    LOGV("track %d format: %s", i, s);
    const char *mime;
    if (!AMediaFormat_getString(format, AMEDIAFORMAT_KEY_MIME, &mime)) {
        LOGV("no mime type");
        return JNI_FALSE;
    } else if (!strncmp(mime, "video/", 6)) {
        AMediaExtractor_selectTrack(ex, i);
        decoder = AMediaCodec_createDecoderByType(mime);

        AImageReader* imageReader;
        ANativeWindow* surface;

        // This setting doesn’t works
        media_status_t status = AImageReader_new(480, 360, AIMAGE_FORMAT_RGBA_8888, 1, &imageReader);

        // This setting works
        //media_status_t status = AImageReader_new(480, 360, AIMAGE_FORMAT_YUV_420_888, 1, &imageReader);

        status = AImageReader_getWindow(imageReader, &surface);

        // Configure the decoder to render to a surface
        AMediaCodec_configure(codec, format, surface, NULL, 0);

        AMediaCodec_start(codec);
    }

    AMediaFormat_delete(format);
}

Elsewhere, here below is how I am setting up the callback for the ImageReader:

AImageReader_ImageListener* imageListener = new AImageReader_ImageListener();
imageListener->onImageAvailable = &imageCallback;
AImageReader_setImageListener(imageReader, imageListener);

And finally, here below is how the callback looks like:

void imageCallback(void *context, AImageReader *reader) {

    int32_t format;
    media_status_t status = AImageReader_getFormat (reader, &format);

    AImage* image;
    status = AImageReader_acquireLatestImage(reader, &image);

    status = AImage_getFormat(image, &format);

    // further processing to follow
    ...
}

The issue that I have been facing is that if I configure the ImageReader with RGBA format, the image in the callback always comes out to be NULL:

// Always NULL for ImageReader configured with RGBA
// OK for ImageReader configured with YUV_420_888
AImage* image;
status = AImageReader_acquireLatestImage(reader, &image);

Am I using the NDK API correctly here? One thing I would like to mention is that the RGBA doesn't appears in the list of decoder capabilities as fetched via the following API (not provided via NDK, tried it in the Java layer):

 getCodecInfo().getCapabilitiesForType(…).colorFormats

Upvotes: 2

Views: 1333

Answers (1)

mstorsjo
mstorsjo

Reputation: 13317

Video decoders normally don't support outputting in RGB format (as you noticed in the colorformats codec info), so that's why this won't work.

The video decoder output can be transparently converted into RGB if you use a surface texture as the output for the decoder - then the decoded video data is available within an OpenGL context. If you then just do a plain 1:1 copy/rendering of the surface texture and have the OpenGL context set up to render into an ImageReader, I would expect you to get the RGB data you need.

It is a bit roundabout (and I'm not sure how easily accessible all the APIs are in a native code context), but should be doable as far as I know.

Upvotes: 1

Related Questions