Finding outlier points of a curve

Question

I have a range of points that I get after implementing contour detection on thousands of images. It is expected to be like a smooth spline. But sometimes algorithm fails due to quality of the image. Main problem is that, curve got dropping or jumping points. I present examples:

This is good/expected result:

Curve goes well then after x = 500 it went wrong:

Here is a big gap to avoid:

I am trying to avoid this gap and do a great estimation with spline. But sometimes gap is too big that they effect the spline.

My question is how can I provide a solution to avoid these outliers(even though they are too many sometimes) and find the best estimation of curve?

I tried outlier detection with z-score and IQR but sometimes these drops are too big that they change the average values so it cannot be detected with z-score.

On the other hand, spline doesn't work as gap is too big sometimes.

Another try was to find average slope of the curve then remove those who are far from it enough. But this lead to removing correct points. One can expect removing points after x=500 in the 3rd example I provided earlier.