CODEWITHSUNDEEP
machine-learningneural-networkdeep-learningcaffe
Aleksander Monk
Aleksander Monk

Reputation: 2907

Caffe, joining outputs from 2 models

I have 2 different models, let's say NM1 and NM2.

So, what I'm looking is something that works like in the example below.

Let's say that we have a picture of a dog.

NM1 predicts that it's a cat on the picture with a probability 0.52 and that it's a dog with a probability 0.48. NM2 predicts that it's a dog with a probability 0.6 and that it's a cat with a probability 0.4.

NM1 - will predict wrong NM2 - will predict correctly

NM1 + NM2 - connection will predict correctly (because 0.48 + 0.6 > 0.52 + 0.4)

So, each model ends with InnerProducts (after Softmax) which give me 2 vectors of probabilities.

Next step, I have those 2 vectors and I want to add them. Here I use Eltwise layer. 

layer {
  name: "eltwise-sum"
  type: "Eltwise"
  bottom: "fc8"
  bottom: "fc8N"
  top: "out"
  eltwise_param { operation: SUM }
}

Before joining NM1 had accuracy ~70% and NM2 ~10%.

After joining accuracy can't reach even 1%.

Thus, my conclusion is that I understand something wrong and I'd be grateful if someone could explain to me where I'm wrong.

PS. I did turn off shuffle when creating lmdb.

UPDATE

layer {
  name: "eltwise-sum"
  type: "Eltwise"
  bottom: "fc8L"
  bottom: "fc8NL"
  top: "out"
  eltwise_param { 
  operation: SUM 
  coeff: 0.5
  coeff: 0.5
  }

}


#accur for PI alone
layer {
  name: "accuracyPINorm"
  type: "Accuracy"
  bottom: "fc8L"
  bottom: "label"
  top: "accuracyPiNorm"
  include {
    phase: TEST
  }
}

#accur for norm images alone
layer {
  name: "accuracyIMGNorm"
  type: "Accuracy"
  bottom: "fc8NL"
  bottom: "labelN"
  top: "accuracyIMGNorm"
  include {
    phase: TEST
  }
}

#accur for them together
layer {
  name: "accuracy"
  type: "Accuracy"
  bottom: "out"
  bottom: "label"
  top: "accuracy"
  include {
    phase: TEST
  }
}

Model image

Upvotes: 1

Views: 269

Answers (1)

Shai
Shai

Reputation: 114786

If you want to add (element-wise) the probabilities, you need to add after the "Softmax" layer, and not after the "InnerProduct" layer. You should have something like

layer {
  type: "InnerProduct"
  name: "fc8"
  top: "fc8"
  # ... 
}
layer {
  type: "Softmax"
  name: "prob_nm1"
  top: "prob_nm1"
  bottom: "fc8"
}
layer {
  type: "InnerProduct"
  name: "fc8N"
  top: "fc8N"
  # ... 
}
layer {
  type: "Softmax"
  name: "prob_nm2"
  top: "prob_nm2"
  bottom: "fc8N"
}
# Joining the probabilites
layer {
  type: "Eltwise"
  name: "prob_sum"
  bottom: "prob_nm1"
  bottom: "prob_nm2"
  top: "prob_sum"
  eltwise_param {
    operation: SUM
    coeff: 0.5
    coeff: 0.5
  }
}

Upvotes: 1

Related Questions