Custom Openvino YoloV8 model detecting too many random objects

Question

I trained a Yolo model based on YoloV8s using my own dataset. It can detect two things: person and vehicle.

The model is working fine on a video input if I use the .pt model in python or .onnx model in CPP.

But when I convert it to openvino and run using CPP then I get these weird results after roughly 30 frames where bounding boxes are drawn everywhere.

Also when I run the openvino model with Python I get this error after roughly 20 frames:

video 1/1 (frame 22/100) /home/corentin/Documents/lely/video/random_video/bus_video.mp4: 640x640 208 vehicles, 92 persons, 47.9ms
Traceback (most recent call last):
  File "/home/corentin/Documents/lely/yolo/testYolo/runYoloOnVideo.py", line 23, in 
    frame_with_boxes = result.plot()
  File "/home/corentin/.local/lib/python3.10/site-packages/ultralytics/engine/results.py", line 532, in plot
    annotator.box_label(box, label, color=colors(c, True), rotated=is_obb)
  File "/home/corentin/.local/lib/python3.10/site-packages/ultralytics/utils/plotting.py", line 314, in box_label
    p1, p2 = (int(box[0]), int(box[1])), (int(box[2]), int(box[3]))
ValueError: cannot convert float NaN to integer
Exception ignored in: 
Traceback (most recent call last):
  File "/home/corentin/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 44, in generator_context
  File "/home/corentin/.local/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 279, in clone
  File "/home/corentin/.local/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 259, in __init__
AttributeError: 'NoneType' object has no attribute '_jit_internal'

I tried different version of openvino without success.

The only place it works fine is in an openvino docker container.

Anyone know what the issue could be ?