How to make HDR video in Python?

Question

The original images are computed values from a simulation.

A video with 12 or at least 10 bits per color is preferred instead of the default 8 bit.

Answer 1

I think now I have find the right way to make a HDR video in python. The only tools we need is openCV and ffmpeg. Firstly convert the frames data into YUVI420, and then write them into a .yuv file, and then use ffmpeg to encode the video.

For example, if you want to make an HDR video, BT.2020, full-range(PC range), 10bit, YUVI420, HLG:

Step 1. Process the frame and convert it into YUV420p10 formation.

Here I give a simple implementation:

def RGB_to_YUV(RGB, gamut="bt2020", bits=10, video_range="full", formation="420"):
    if RGB.dtype == "uint8":
        RGB = RGB / 255.0
    height, width = RGB.shape[:2]

    if bits == 8:
        dtype = np.uint8
    else:
        dtype = np.uint16

    if gamut == "bt709":
        YCbCr = RGB709_to_YCbCr709(RGB)
    elif gamut == "bt2020":
        YCbCr = RGB2020_to_YCbCr2020(RGB)
    else:
        raise Exception("gamut param error!")

    Y = YCbCr[..., 0]
    Cb = YCbCr[..., 1]
    Cr = YCbCr[..., 2]

    if video_range == "limited":
        D_Y = np.clip(np.round(Y * 219 + 16), 16, 235).astype(dtype) * np.power(2, bits - 8)
        D_Cb = np.clip(np.round(Cb * 224 + 128), 16, 240).astype(dtype) * np.power(2, bits - 8)
        D_Cr = np.clip(np.round(Cr * 224 + 128), 16, 240).astype(dtype) * np.power(2, bits - 8)

    elif video_range == "full":
        D_Y = np.clip(np.round(Y * 255), 0, 255).astype(dtype) * np.power(2, bits - 8)
        D_Cb = np.clip(np.round(Cb * 254 + 128), 1, 255).astype(dtype) * np.power(2, bits - 8)
        D_Cr = np.clip(np.round(Cr * 254 + 128), 1, 255).astype(dtype) * np.power(2, bits - 8)

    else:
        raise Exception("param: video_range error!")

    y_size = height * width
    uv_size = height // 2 * width // 2
    frame_len = y_size * 3 // 2

    if formation == "420":
        U = cv.resize(D_Cb, None, None, 0.5, 0.5).flatten()
        V = cv.resize(D_Cr, None, None, 0.5, 0.5).flatten()

        yuv = np.empty(frame_len, dtype=dtype)
        yuv[:y_size] = D_Y.flatten()
        yuv[y_size: y_size + uv_size] = U
        yuv[y_size + uv_size:] = V
        return yuv

    elif formation == "444":
        Y = D_Y
        U = D_Cb
        V = D_Cr
        return cv.merge((Y, U, V))


def RGB709_to_YCbCr709(RGB):
    if RGB.dtype == "uint8":
        RGB = RGB / 255.0

    m_RGB709_to_YCbCr709 = np.array([[0.21260000, 0.71520000, 0.07220000],
                                     [-0.11457211, -0.38542789, 0.50000000],
                                     [0.50000000, -0.45415291, -0.04584709]])

    return np.matmul(RGB, m_RGB709_to_YCbCr709.T)


def RGB2020_to_YCbCr2020(RGB):
    m_RGB2020_to_YCbCr2020 = np.array([[0.26270000, 0.67800000, 0.05930000],
                                       [-0.13963006, -0.36036994, 0.50000000],
                                       [0.50000000, -0.45978570, -0.04021430]])

return np.matmul(RGB, m_RGB2020_to_YCbCr2020.T)

Here, the RGB frame you input to this function should be a non-linear RGB, which means, it is "un-gammaed". So, if you want to see it with the OpenCV imshow(), it may goes some strange with low contrast, gray-yellow tonal.

Step 2. Write these YUV420 data into a ".yuv" file.

def writeYUVFile(path, data):
    with open(path, "ab") as f:
        f.write(data.tobytes())

Step 3. Use ffmpeg to encode this yuv file into an hdr video

ffmpeg -s 1080x1920 -r 30 -pix_fmt yuv420p10le -i yuv/file/path.yuv -c:v libx265 -tag:v hvc1 -x265-params "colorprim=bt2020:transfer=arib-std-b67:colormatrix=bt2020nc:range=full" -pix_fmt yuv420p10le -r 30 output/file/path.mov -y

Of course, you can change the parameters to meet your need. And to find more details about x265-params, please see https://x265.readthedocs.io/en/master/cli.html

Last but not least, I must say that, in such way you can make an HDR video in Python, but I don't recommend to do it in this way for openCV's preview feature(imshow()) doesn't support the HDR preview and BT.2020 color gamut(or maybe I was wrong, if there is some way to preview HDR video with OpenCV, please tell me). Generally speaking, I think it will demonstrate a picture with sRGB color gamut, which is vary different from the BT.2020 or P3 color gamut we usually used to make HDR videos. It means you cannot control the effect real-time. You must generate the video, and check it, then adjust the param, and repeat. Believe me, this process is really boring and, hard to find the best params.

I really hope this answer will help, cause I met the same question when I was a rookie in this field.

Answer 2

I got two ideas (which has not been verified).

The first is simple but feasible, that is, write those frames as .exr files which contain hdr data. And then use ffmpeg or lumahdrv to generate an HDR video.

The second may be easier but I'm not sure if it will work. In openCV, the class VideoWriter has an attribute cv.VIDEOWRITER_PROP_DEPTH, which is CV_8U as default, maybe we may set it as CV_16F or CV_32F to write a float frame.

Answer 3

There is a still image solution for Python based on OpenCV based on making several output images with different intensities. Can the same techniques be used for video by creating several output videos and overlapping them?