CODEWITHSUNDEEP
videoffmpegmp4h.264ms-media-foundation
Jules
Jules

Reputation: 671

How do I extract color matrix from MP4 an x264 stream in Media Foundation

I am playing a video (mp4 containing x264 encoded video stream) with a custom player using media foundation.

When I convert the YUV information into RGB I need to account for the color matrix and range used at encode time.

Some of my videos have this information, I can use MediaInfo.exe or FFMPEG to see that it is present.

However, for such videos if I look at the relevant Media Foundation properties (Extended Color Information) the properties are not present in the files.

So, somehow I need to find a way to access the information.

Media Foundation does provide access to MF_MT_MPEG4_SAMPLE_DESCRIPTION and MF_MT_MPEG_SEQUENCE_HEADER for the video stream but I can't find descriptions of what these contain.

I noticed that the MF_MT_MPEG_SEQUENCE_HEADER is much longer for the videos with the information present and this (MPEG Headers Quick Reference) seems to suggest headers might contain the information I need.

I'm looking for Color Range (limited/full), Color Primaries, Transfer Characteristics and Matrix Coefficients (BT.709 etc).

I'd greatly appreciate any help finding this information from a Media Foundation video stream.

Thanks

Jules

Update - Sequence Header

The sequence header appears to be a subset of MPEG4 sample description, though I can't find anything that indicates what either bits of data actually contains / doesn't contain specifically.

The sequence header appears to contain data structured as an MP4 byte stream as described in the H264 Standards Document and includes the VUI (Video Usability Information - Annex E of document) which may then include the colour information I'm interested in.

Given that it's a byte stream I need to know where it starts and whether there's some existing code I could use to decode it.

In FFMPEG in libavcodec/h264_ps.c there is a function called ff_h264_decode_seq_parameter_set which ends up calling decode_vui_parameters. It seems possible that seq_parameter_set maps to MF_MT_MPEG_SEQUENCE_HEADER and it may be possible to use that code to decode the data.

If anyone one has any direct experience with decoding this data it would be very useful.

Thanks again

Update - Related posts

I found this How to decode sprop-parameter-sets in a H264 SDP? and Possible Locations for Sequence/Picture Parameter Set(s) for H.264 Stream which are fairly helpful.

The sequence header would appear to be Sequence or picture parameter set (pps) and the parameters I want are the VUI extension subset.

Plus this post H.264 stream structure gives the high level of how the stream data is structured, and the MF_MT_MPEG_SEQUENCE_HEADER appears to start with a NAL 0x00 0x00 0x01 so I'm guessing it is a NAL containing the PPS.

Upvotes: 4

Views: 2139

Answers (2)

mofo77
mofo77

Reputation: 1515

Yes, the information is in the VUI extension subset.

So you need to check for vui_parameters_present_flag and video_signal_type_present_flag. If those flags are not set, i think there are default values for Color Range/Color Primaries/Transfer Characteristics/Matrix Coefficients in the h264 specifications.

Also you need to extract information from the SPS and what you have to care to do this, is to learn about Exponential-Golomb coding.

Check this link : The h.264 Sequence Parameter Set

EDIT

Also yes, MF_MT_MPEG_SEQUENCE_HEADER normaly contains SPS and PPS information, when file is h264 video format.

Upvotes: 1

Roman Ryltsov
Roman Ryltsov

Reputation: 69706

(a cross-post from MSDN Forums)

You can look up sample code parsing H.264 parameter sets in GraphStudioNext here in H264StructReader.cpp, as well as other projects with available code.

GraphStudioNext will also be able to provide this information (colour_primaries etc.) visually, where it's available:

Image above displays a GraphStudioNext window with visualization of H.264 media type (for a MP4 file with respective H.264 data), which embeds parameter sets, which in turn (for that specific file) contain a non-empty section with color primaries and matrix coefficients.

Indeed, even though the values are present in the parameter set, Media Foundation does not make them available in the media type (MPEG-4 File Source does not add respective attributes, initialized from the parameter sets).

Upvotes: 2

Related Questions