How to correctly encrypt the I-frames in video files?

Question

I'm trying to encrypt only the I-frames of a video file so that the energy overhead is lessened compared to if I tried to encrypt the whole file. However, after encrypting what I thought were the I-frames, the video seems to play normally (except for the encrypted frames). I thought that the point of encrypting the I-frames was so that the other frames would be distorted, but that doesn't seem to be the case. I was wondering what seems to be wrong ? The file format is avi with codec MPEG-4 video (FMP4).

I used ffmpeg to identity the I frames of a certain video (named "video1") : "ffprobe -select_streams v -show_frames -of csv video1.avi > video_frame_info.csv". I got this table which seems to indicate that every 12th frame is an I-frame. I then used this python program to encrypt every 12th frame with chacha20 from the pycryptodome library :

from Crypto.Cipher import ChaCha20
from Crypto.Random import get_random_bytes
import cv2
import numpy as np

def read_frames(video_path):
    # Load the video file
    video = cv2.VideoCapture(video_path)

    # Initialize a list to hold frames
    frames = []

    # Loop until there are frames left in the video file
    while video.isOpened():
        ret, frame = video.read()
        if not ret:
            break

        frames.append(frame)

    video.release()

    return frames

def encrypt_iframes(frames, key):
    # Initialize the cipher
    cipher = ChaCha20.new(key=key)

    # Encrypt every 12th frame (considering the first frame as an I-frame)
    for i in range(0, len(frames), 12):
        frame = frames[i]
        # Flatten and bytes-encode the frame
        flat_frame = frame.flatten()
        bytes_frame = flat_frame.tobytes()

        # Encrypt the frame
        encrypted_frame = cipher.encrypt(bytes_frame)

        # Replace the original frame with the encrypted one
        frames[i] = np.frombuffer(encrypted_frame, dtype=flat_frame.dtype).reshape(frame.shape)

    return frames

def write_frames(frames, output_path):
    # Assume all frames have the same shape
    height, width, _ = frames[0].shape

    # Create a VideoWriter object
    out = cv2.VideoWriter(output_path, cv2.VideoWriter_fourcc(*'mpv4'), 30, (width, height))

    for frame in frames:
        out.write(frame)

    out.release()

def main():
    # Generate a random key
    key = get_random_bytes(32)

    # Read frames from video
    frames = read_frames('video1.avi')

    # Encrypt I-frames
    encrypted_frames = encrypt_iframes(frames, key)

    # Write encrypted frames to a new video file
    write_frames(encrypted_frames, 'reprise.avi')

if __name__ == "__main__":
    main()

When I play the video, every 12th frame there is just noise on the screen, which makes sense since it was encrypted. However, all the other frames seem to be completely normal, even thought they are P-frames and should depend on the I-frames. What am I doing wrong, and how can I make it so that it works as intended ? Thanks in advance