CODEWITHSUNDEEP
matlabmachine-learninglogistic-regressiongradient-descent
vega
vega

Reputation: 2189

Why does the Logistic Regression cost go negative and not correct?

I am implementing logistic regression in Matlab. The data is normalized (mean and std). I understand that depending on your learning rate you may overshoot the optimal point. But doesn't that mean your cost starts going up? In my case the cost goes into negative territory, I don't understand why.

Here is the standard (I think?) cost and weight update rule

function J = crossEntropyError(w, x, y)
  h = sigmoid(x*w);
  J = (-y'*log(h) - (1-y')*log(1-h));
end

Weight update:

function w = updateWeights(alpha, w, x, y)      
  h = sigmoid(x*w);
  gradient = x'*(h-y);
  w = w - alpha*gradient;
end

This is what happens with my cost, x-axis is the iteration: logistic regression cost

This makes no sense. When hitting 0, isn't it supposed to self-correct and go in the other direction? That is, since the derivative points to the minimum. I've played with the learning rate, here it's set to a tiny 0.0001. But it makes no difference, same pattern. What's the issue? There must be something really wrong here, I can't find it though.

Upvotes: 1

Views: 1762

Answers (2)

vega
vega

Reputation: 2189

So I realized my mistake, it's quite silly. I was using a dataset where the labels are not boolean (0 or 1) which resulted in the cross entropy error above. The code was correct, but not suitable for labels with non-boolean data.

I would delete the question but I wouldn't want my account to be blocked. Maybe it can help someone?

Upvotes: 2

Bilal
Bilal

Reputation: 11

Please try this cost function 

J = -1/(m)*sum(y.*log(sigmoid(x*w))+(1-y).*log(1-sigmoid(x*w);

as

m = sieze(x)

and 

gradient = zeros(size(theta)); 

 for i = 1:size(theta),
     gradient(i)= (1/m)*sum((sigmoid(X*theta)-y).*X(:,i));

 end;

Upvotes: 0

Related Questions