From an array of points, select the most inner ones towards a given line

Question

I have a set of points (xs, ys) and a 2-degree polynomial.

import numpy as np
import matplotlib.pyplot as plt

# NOTE: these are already sorted by x
xs = np.array([12.88878822, 16.16217041, 17.28821182, 22.57517242, 25.35802078, 28.0203743, 29.80039215, 31.26796722, 33.99120331, 35.18927765, 37.00539398, 38.59598541, 38.59766006, 38.95425415, 41.65776062, 41.66015625, 43.88010788, 44.61865234, 46.34718704, 48.43397903, 48.85764694, 50.84202957, 53.73672104, 54.67554474, 56.78898239, 57.91404724, 58.72961807, 59.45791626, 60.3879776, 60.78292465, 64.89099121, 65.02430725, 66.80397034, 68.67731476, 69.57426453, 70.65918732, 72.58442688, 76.2591629, 76.5898819, 78.45317841, 78.7919693, 80.52996063, 80.63969421, 82.64718628, 84.55006409, 85.319664, 86.59933472, 86.68675995, 88.05833435, 94.50637817, 98.40599823, 100.93874359, 102.75447083, 105.96748352, 107.09364319, 109.96869659, 111.82051849, 114.35180664, 115.90748596, 118.63829803, 122.25462341, 127.58323669, 138.77226257, 158.59503174])
ys = np.array([11.75947666, 11.74938011, 12.04140949, 11.82363796, 12.0559206, 12.14261723, 11.81262589, 12.25062943, 12.34748745, 12.07851601, 12.64596748, 3.45692825, 5.1812706, 5.23445797, 3.22935534, 12.34285259, 2.85700059, 12.83814716, 3.03312325, 13.86905003, 2.73906446, 2.36633062, 2.16316056, 13.695261,  2.11045575, 13.76149082, 1.80426383, 13.94807529, 1.26637924, 13.85880852, 1.15412867, 14.43988228, 0.97052693, 0.77311337, 14.30804253, 0.79675227, 0.3398909, 14.97122669, 0.35970068, -0.41222754, 14.95026112, -0.69134873, 14.76232147, -0.71044016, -1.29101408, 15.07320309, 14.71334267, -1.61440754, 14.36712933, -2.3999927, 15.45431614, -2.90943909, 16.16511154, 15.59430504, -3.45461845, -3.55082941, 15.71772671, -5.24281168, 17.1123867, -6.24958754, 19.35256958, -8.47069263, -9.70999622, -12.22311211])

# draw line from which to view
curvature = -0.0015640259
curve_x = np.linspace(np.min(xs - 10), np.max(xs + 10), 200)
curve_y = (0.5 * curvature) * curve_x * curve_x + (np.sin(-curvature) * curve_x)

fig, ax = plt.subplots()
fig.set_size_inches(5, 8)
plt.xlabel("Ys")
plt.ylabel("Xs")
# xs and ys are reversed intentionally
ax.plot(ys, xs, 'go--', linewidth=2, markersize=12)
ax.plot(curve_y, curve_x, 'r', linewidth=1)

# _sorted_indices = np.argsort(ys)
# sorted_xs = np.array(xs)[_sorted_indices]
# sorted_ys = np.array(ys)[_sorted_indices]
# ax.plot(sorted_ys, sorted_xs, 'r--', linewidth=1)
plt.show()

This produces this plot of points and line:

Now i need to get the points starting from lowest X to highest X (vertical sorting), but if there is another point outside of the previous one, don't add it.

Like so:

Maybe I'm tired, but I cannot figure out a simple way to do this.
NOTE that the Y values go into negative, so this needs to work for +/- Ys.
Any quick obvious solution I'm missing?