CODEWITHSUNDEEP
neural-networkdeep-learningcaffepycaffe
zbqv
zbqv

Reputation: 79

Modify threshold in ReLU in Caffe framework

I am new to Caffe, and now I need to modify the threshold values in ReLU layers in a convolution neural network. The way I am using now to modify thresholds is to edit the C++ source code in caffe/src/caffe/layers/relu_layer.cpp, and recompile it. However, this will change the threshold value to a specified value every time when ReLU is called. Is there any way to use different value as threshold in each ReLU layers in a network? By the way, I am using the pycaffe interface and I cannot find such a way to do so.

Finally, sorry for my poor English, if there are something unclear, just let me know, I'll try to describe it in detail.

Upvotes: 4

Views: 798

Answers (2)

Shai
Shai

Reputation: 114866

If I understand correctly your "ReLU with threshold" is basically

f(x) = x-threshold if x>threshold, 0 otherwise

You can easily implement it by adding a "Bias" layer which subtract threshold from the input just prior to a regular "ReLU" layer

Upvotes: 3

Dale
Dale

Reputation: 1608

Yes, you can. In src/caffe/proto, add a line:

message ReLUParameter {
  ...
  optional float threshold = 3 [default = 0]; #add this line
  ... 
}

and in src/caffe/layers/relu_layer.cpp, make some small modifications as:

template <typename Dtype>
void ReLULayer<Dtype>::Forward_cpu(const vector<Blob<Dtype>*>& bottom,
    const vector<Blob<Dtype>*>& top) {
  ...
  Dtype threshold = this->layer_param_.relu_param().threshold(); //add this line
  for (int i = 0; i < count; ++i) {
    top_data[i] = (bottom_data[i] > threshold) ? (bottom_data[i] - threshold) : 
                  (negative_slope * (bottom_data[i] - threshold));
  }
}

template <typename Dtype>
void ReLULayer<Dtype>::Backward_cpu(const vector<Blob<Dtype>*>& top,
    const vector<bool>& propagate_down,
    const vector<Blob<Dtype>*>& bottom) {
  if (propagate_down[0]) {
    ...
    Dtype threshold = this->layer_param_.relu_param().threshold(); //this line
    for (int i = 0; i < count; ++i) {
      bottom_diff[i] = top_diff[i] * ((bottom_data[i] > threshold)
          + negative_slope * (bottom_data[i] <= threshold));
    }
  }
}

and similarly in src/caffe/layers/relu_layer.cu the code should be like this.

And after compiling your caffe and pycaffe, in your net.prototxt, you can write a relu layer like:

layer {
  name: "threshold_relu"
  type: "ReLU"
  relu_param: {threshold: 1 #e.g. you want this relu layer to have a threshold 1}
  bottom: "input"
  top: "output"
}

Upvotes: 3

Related Questions