Where does the graph of the loss function in machine learning come from?

Question

Where does the graph of the loss function in machine learning come from? I am studying about machine learning. I sometimes don't understand models that have been optimized using regularization terms. In the explanation of regularization, the following figure may appear. Here is an example of the L1 regularization term. I have assumed that the model has two weight parameters w1, w2. That is, the equation of model y is expressed by the following equation.

y = w1x1 + w2x2

For simplicity, I ignored the bias term.

The red squares represent regularization terms. And the blue ellipses are represents the loss function without the regularization term. The regularization term is given by

| w1 | ^ q + | w2 | ^ q = r ^ q (r is const.)

Therefore, the equation of the graph at w1> 0 and w2> 0 is expressed as follows.

w2 = (r ^ q-| w1 | ^ q) ^ (1 / q)

By substituting w1 for this equation (q = 0 at Lasso), you can draw a graph of the regularized term.

On the other hand, I could not draw a graph of the loss function. Perhaps you need more than one piece of data to draw this graph. For simplicity, I have assumed that I have only two pieces of data. I define them as (x11, x12, t1), (x21, x22, t2). When the loss function is MSE, it is expressed by the following equation.

Ed = 1/2 * {(t1-w1x11-w2x12) + (t1-w1x21-w2x22)}

If I simplify this, it is expressed as

Ed = a*w1^2 + b*w1 + c*w2^2 + d*w2 + e*w1*w2 + f

Here, a, b, c, d, e, and f are functions represented by all or part of x11, x12, x21, and x22. After finding a, b, c, d, e, and f, I thought that if we substitute w1 for this equation, we could draw a graph of the loss function. However, I cannot draw well. Is the above understanding correct? Thank you.

Answer 1

To visualize the loss function, Ed which is a function of w1 and w2, we should visualize it as a 3 dimensional plot. For example, you can use Geogebra to visualize a 3 dimensional surface plot.

Here is an example, where a=3, b=-1, c=1, d =-1 , e=2.

The 2D plot that you see is called a countor plot. This link enables you to draw it online.

To draw a contour plot manually, you fix the value of Ed, then you obtained a quadratic equation, after which, as you varies w1, you can solve for your w2, for each w1, you can obtain up to 2 w2 as it is quadratic.

Remark: If you are looking for closed form expression in terms of arbitrary q, that could be more challenging.