FFmpeg and OpenCV: Issues with RTSP Stream Decoding from H.265 Camera (SIYI A8 mini gimbal camera)

Question

I am working on an object detection and tracking project using a camera that streams H.265 video over RTSP (rtsp://192.168.144.25:8554/main.264. However, I am encountering issues with both FFmpeg, ffplay, VLC and all other media players while decoding the stream.

All the media players are having glitch in the frame (packet loss). I am using Ubuntu 20.04 but my final deployment will be in Jetson Xavier NX. Both are having this problem. I am new to this field, Kindly give your suggestions.

Commands used are:

aadarsh@aadarsh-B560M:~/Desktop$ sudo ip link set enp3s0 up
[sudo] password for aadarsh: 
aadarsh@aadarsh-B560M:~/Desktop$ sudo ethtool enp3s0
aadarsh@aadarsh-B560M:~/Desktop$ sudo ethtool enp3s0
Settings for enp3s0:
    Supported ports: [ TP MII ]
    Supported link modes:   10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Full 
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Full 
    Advertised pause frame use: Symmetric Receive-only
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: 10Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: off
    MDI-X: Unknown
    Supports Wake-on: pumbg
    Wake-on: d
    Link detected: yes
        aadarsh@aadarsh-B560M:~/Desktop$ ping 192.168.144.25
        PING 192.168.144.25 (192.168.144.25) 56(84) bytes of data.
        64 bytes from 192.168.144.25: icmp_seq=1 ttl=64 time=5.56 ms
        64 bytes from 192.168.144.25: icmp_seq=2 ttl=64 time=2.82 ms
        64 bytes from 192.168.144.25: icmp_seq=3 ttl=64 time=0.386 ms
        64 bytes from 192.168.144.25: icmp_seq=4 ttl=64 time=0.889 ms
        64 bytes from 192.168.144.25: icmp_seq=5 ttl=64 time=1.47 ms
        64 bytes from 192.168.144.25: icmp_seq=6 ttl=64 time=0.869 ms
        64 bytes from 192.168.144.25: icmp_seq=7 ttl=64 time=0.770 ms
        64 bytes from 192.168.144.25: icmp_seq=8 ttl=64 time=0.378 ms
        64 bytes from 192.168.144.25: icmp_seq=9 ttl=64 time=0.408 ms

wWhat I tried:

1.Using FFmpeg to decode the stream:

Expected: Smooth playback and decoding of the H.265 RTSP stream without frame drops or errors.

Result: Frequent errors like Could not find ref with POC, RTP: missed packets, and occasional process termination due to resource exhaustion.

2. OpenCV for real-time processing:

   Expected: Continuous frame capture with minimal latency for further processing using YOLOv8. Result: Slow frame processing, frequent frame drops, and delayed responses.

3. Increasing buffer size and using hardware acceleration:

   Expected: Reduced frame drops and improved decoding stability. Result: Minor improvement, but errors and instability persist.

4. Testing with VLC and FFplay:

   Expected: Consistent playback of the RTSP stream. Result: VLC and FFplay show similar issues with "Waiting for VPS/SPS/PPS"