CODEWITHSUNDEEP
pythondeep-learningcntk
OmG
OmG

Reputation: 18858

Alternatives for loss functions in python CNTK

I have created a sequential model in CNTK and pass this model into a loss function like the following:

ce = cross_entropy_with_softmax(model, labels)

As mentioned here and as I have multilabel classifier, I want to use a proper loss function. The problem is I can not find any proper document to find these loss functions in Python. Is there any suggestion or sample code for this requirement.

I should notice that I found these alternatives (logistic and weighted logistic) in BrainScript language, but not in Python.

Upvotes: 2

Views: 855

Answers (3)

Frank Seide MSFT
Frank Seide MSFT

Reputation: 491

"my data has more than one label (three label) and each label has more than two values (30 different values)"

Do I understand right, you have 3 network outputs and associated labels, and each one is a 1-in-30 classifier? Then it seems you can just add three cross_entropy_with_softmax() values. Is that what you want?

E.g. if the model function returns a triple (ending in something like return combine([z1, z2, z3])), then your criterion function that you pass to Trainer could look like this (if you don't use Python 3, the syntax is a little different):

from cntk.layers.typing import Tensor, SparseTensor
@Function
def my_criterion(input : Tensor[input_dim], labels1 : SparseTensor[30],
                 labels2 : SparseTensor[30], labels3 : SparseTensor[30]):
    z1, z2, z3 = my_model(input).outputs
    loss = cross_entropy_with_softmax(z1, labels1) + \
           cross_entropy_with_softmax(z2, labels2) + \
           cross_entropy_with_softmax(z3, labels3)
    return loss

learner = ...
trainer = Trainer(None, my_criterion, learner)

# in MB loop:
input_mb, L1_mb, L2_mb, L3_mb = my_next_minibatch()
trainer.train_minibatch(my_criterion.argument_map(input_mb, L1_mb, L2_mb, L3_mb))

Upvotes: 1

Nikos Karampatziakis
Nikos Karampatziakis

Reputation: 2050

Update (based on comments below): If you are using a sequential model then you are probably interested in taking a sum over all positions in the sequence of the loss at each position. cross_entropy_with_softmax is appropriate for the per-position loss and CNTK will automatically compute the sum of the loss values over all positions in the sequence.

Note that the terminology multilabel is non-standard here as it is typically referring to problems with multiple binary labels. The wiki page you link to refers to that case which is different from what you are doing.

Original answer (valid for the actual multilabel case): You will want to use binary_cross_entropy or weighted_binary_cross_entropy. (We decided to rename Logistic when porting this to Python). At the time of this writing these operations only support {0,1} labels. If your labels are in (0,1) then you will need to define your loss like this 

import cntk as C
my_bce = label*C.log(model)+(1-label)*C.log(1-model)

Upvotes: 1

wr.
wr.

Reputation: 2859

Currently, most operators are in the cntk.ops package and documented here. The only exception being the sequence related operators, which reside in cntk.ops.sequence.

We have plans to restructure the operator space (without breaking backwards compatibility) to increase discoverability.

For your particular case, cross_entropy_with_softmax seems to be a reasonable choice, and you can find its documentation with examples here. Please also check out this Jupyter Notebook for a complete example.

Upvotes: 0

Related Questions