GStreamer increase RTSP buffering

Question

I'm trying to use GStreamer to read an RTSP stream. I'm using this pipeline:

gst-launch-1.0 \
    rtspsrc location=<url> latency=2000 ! \
    queue ! \
    rtph264depay ! h264parse ! avdec_h264 ! \
    videoconvert ! videoscale ! \
    autovideosink

The image appears but every ~3s it freezes for ~3s showing artifacts, then works again, and so on. It looks like Gstreamer is not buffering enough, but increasing the latency parameter does not fixes the problem.

In the end I want to send the video to a DeckLink card using decklinkvideosink. When doing so I have the same behavior, and I have to add a rtpjitterbuffer element after my rtspsrc, otherwise I get a black frame.

What could be wrong?