CODEWITHSUNDEEP
pythonmathstatisticsmachine-learningnumerical-methods
Legend
Legend

Reputation: 116810

Finding the elbow point of a curve in a stable way?

I am aware of the existence of this, and this on this topic. However, I would like to finalize on an actual implementation in Python this time.

My only problem is that the elbow point seems to be changing from different instantiations of my code. Observe the two plots shown in this post. While they appear to be visually similar, the value of the elbow point changed significantly. Both the curves were generated from an average of 20 different runs. Even then, there is a significant shift in the value of the elbow point. What precautions can I take to make sure that the value falls within a certain bound?

My attempt is shown below:

def elbowPoint(points):
  secondDerivative = collections.defaultdict(lambda:0)
  for i in range(1, len(points) - 1):
    secondDerivative[i] = points[i+1] + points[i-1] - 2*points[i]

  max_index = secondDerivative.values().index(max(secondDerivative.values()))
  elbow_point = max_index + 1
  return elbow_point

points = [0.80881476685027154, 0.79457906121371058, 0.78071124401504677, 0.77110686192601441, 0.76062373158581287, 0.75174963969985187, 0.74356408965979193, 0.73577573557299236, 0.72782434749305047, 0.71952590556748364, 0.71417942487824781, 0.7076502559300516, 0.70089375208028415, 0.69393584640497064, 0.68550490458450741, 0.68494440529025913, 0.67920157634796108, 0.67280267176628761]
max_point = elbowPoint(points)

enter image description here enter image description here

Upvotes: 4

Views: 3396

Answers (1)

jk.
jk.

Reputation: 14004

Its sounds like your actual concern is how to smooth your data as it contains noise? in which case perhaps you should fit a curve to the data first, then find the elbow of the fitted curve?

Whether this will work would depend on the source of the noise, and if the noise is important for your application? by the way you may want to see how sensitive your fit is to your data by seeing how it changes (or hopefully doesn't) when a point is omitted from the fit (obviously with a high enough polynomial you will always get a good fit to a specific set of data, but you are presumably interested in the general case)

I have no idea if this approach is acceptable, intuitively though i'd think that sensitivity to small errors is bad. ultimately by fitting a curve you are saying that the underlying process is, in the ideal case, modelled by the curve, and any deviation from the curve is an error/noise

Upvotes: 3

Related Questions