CODEWITHSUNDEEP
javascriptraspberry-pivideo-streamingh.264decoding
Tim Perry
Tim Perry

Reputation: 13216

H.264 decoding failing to parse unusual frame headers

I'm reading an H.264 stream from a camera on a raspberry pi. I'm trying to pass this to Broadway via websockets to render in a web page.

The stream contains NAL units, and I'm chunking it on the [0,0,0,1] start prefix code, to send and then decode NAL units individually. I think that's working fine, but Broadway can't decode the result I end up with.

Digging into the parsing code I've based this on though, it seems to be expecting the 5th byte (straight after the start prefix code) to be either:

I've seen a lot of mention of these elsewhere too. All the units I have coming through though seem to start with (in order):

What do these headers mean in an H.264 stream? Do they suggest something about what I need to do to match Broadway's expectations?

(If the full code would be useful to understand this better, see https://github.com/pimterry/pi-cam/tree/d801de9b)

Upvotes: 1

Views: 1848

Answers (1)

Tim Perry
Tim Perry

Reputation: 13216

This was a red herring: the actual issue for me here was that some existing frame dropping logic meant that I wasn't passing Broadway the first few frames in the stream, and it was failing to render. Replaying the SPS and PPS frames for all new connections and making sure they're never dropped has fixed the issue nicely.

I also did work out what these bytes are though, which helped, and may be useful for others for reference:

Hex    Binary     NAL type  Meaning
0x65 = 11 00101 = type 5    Coded slice of an IDR picture (I-frame)
0x41 = 10 00001 = type 1    Coded slice of a non-IDR picture (P-frame)

0x27 = 01 00111 = type 7    Sequence parameter set (B-frame)
0x28 = 01 01000 = type 8    Picture parameter set (B-frame)
0x25 = 01 00101 = type 5    Coded slice of an IDR picture (B-frame)
0x21 = 01 00001 = type 1    Coded slice of a non-IDR picture (B-frame)

Special thanks to Jaromanda X though - the NAL units article [here] and the nal_ref_idc article made working this out much easier.

Upvotes: 4

Related Questions