What is MediaCodec, MediaExtractor and MediaMuxer in android?

Question

What does MediaCodec, MediaExtractor and MediaMuxer mean in android? I am not a video person but I do know what encoding and decoding means, at a basic level. I need to know what are the functions of each classes and at which use cases are they used.
I would also like to know:

• If I want to extract frames from a camera preview and create a video file along with some editing (like speed), which classes should I use and how does it work together?
• If I want to create a Video Player like Exoplayer (not all the functions but a simple Dash adaptive streaming player) , which classes should I use and how does it work together?

Hope you will answer. Thank You.

Answer 1

Let me start of by saying that it is hard to understand this API`s if you don't understand how video encoding/decoding works. I would suggest doing research about how encoders/decoders work.

I will provide an oversimplified explanation of each.

---

MediaCodec:

MediaCodec class can be used to access low-level media codecs, i.e. encoder/decoder components. It is part of the Android low-level multimedia support infrastructure

So MediaCodec handles the decoding or encoding of the video packets/buffers and is responsible for the interaction with the codec.

Here is an example of how to Initialize MediaCodec:

// Create Mediacodec instance by passing a mime type. It will select the best codec for this mime type
MediaCodec mDecoder = MediaCodec.createDecoderByType(mimeType); 
// Pass an instance on MediaFormat and the output/rendering Surface
mDecoder.configure(format, surface, null, 0); 
mDecoder.start();

You would then start passing buffers to MediaCodec, like this:

ByteBuffer[] inputBuffers = mDecoder.getInputBuffers();
int index = mDecoder.dequeueInputBuffer(timeout);
// Check if buffers are available
if (index >= 0) { 
    // Get dequeued buffer
    ByteBuffer buffer = inputBuffers[index]; 
    // Get sample data size to determine if we should keep queuing more buffers or signal end of stream
    int sampleSize = mExtractor.readSampleData(buffer, 0); 
    if (sampleSize < 0) { 
        // Signal EOS, this happens when you reach the end if the video, for example.
        mDecoder.queueInputBuffer(inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM); 
    } else { 
        // Queue the dequeued buffer and pass the extractors sample time
        mDecoder.queueInputBuffer(index, 0, sampleSize, mExtractor.getSampleTime(), 0);
        mExtractor.advance(); 
    } 
}

You then dequeue the output buffer and release it to your surface:

BufferInfo frameInfo = new BufferInfo(); 
int index mDecoder.dequeueOutputBuffer(frameInfo, timeout);

switch (index) { 
    case
        MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED: 
    break; 
    case
        MediaCodec.INFO_OUTPUT_FORMAT_CHANGED: 
        MediaFormat newFormat = mDecoder.getOutputFormat(); 
    break; 
    case
        MediaCodec.INFO_TRY_AGAIN_LATER: break; default: 
    break; 
}

// You can now push the frames to the surface
// This is where you can control the playback speed, you can do this by letting your thread sleep momentarily

if (index > 0) {
    mDecoder.releaseOutputBuffer(bufferIndex, true);
}

---

MediaExtractor:

MediaExtractor facilitates extraction of demuxed, typically encoded, media data from a data source.

The documentation description is self-explanatory.

Have a look below, I've added comments to make it more understandable:

// Initialize the extractor
MediaExtractor() mExtract = new MediaExtractor(); mExtract.setDataSource(mSource);

// Select/set the video track (if available)
int trackIndex = selectVideoTrack(mExtract);
if(trackIndex < 0) 
    throw new IOException("Can't find track");
mExtract.selectTrack(trackIndex);

// The extractor is now ready to be used

// Get the track format
mFormat = mExtractor.getTrackFormat(trackIndex);

// Get buffer size to check if a buffer is available
// This will be used by MediaCodec to determine if buffers are available
sampleSize = mExtractor.readSampleData(buffer, 0);

---

MediaMuxer:

MediaMuxer facilitates muxing elementary streams. Currently MediaMuxer supports MP4, Webm and 3GP file as the output. It also supports muxing B-frames in MP4 since Android Nougat.

This is self-explanatory once again. It's used to create a video/audio file. For example, merging two video files together.