Faster-RCNN, why don't we just use only RPN for detection?

Question

As we know, faster-RCNN has two main parts: one is region proposal network(RPN), and another one is fast-RCNN.

My question is, now that region proposal network(RPN) can output class scores and bounding boxes and is trainable, why do we need Fast-RCNN?

Am I thinking it right that the RPN is enough for detection (red circle), and Fast-RCNN is now becoming redundant (blue circle)?

Answer 1

If you just add a class head to the RPN Network, you would indeed get detections, with scores and class estimates.

However, the second stage is used mainly to obtain more accurate detection boxes.

Faster-RCNN is a two-stage detector, like Fast R-CNN. There, Selective Search was used to generate rough estimates of the location of objects and the second stage then refines them, or rejects them.

Now why is this necessary for the RPN? So why are they only rough estimates?

One reason is the limited receptive field: The input image is transformed via a CNN into a feature map with limited spatial resolution. For each position on the feature map, the RPN heads estimate if the features at that position correspond to an object and the heads regress the detection box. The box regression is done based on the final feature map of the CNN. In particular, it may happen that the correct bounding box on the image is larger than the corresponding receptive field due to the CNN.

Example: Lets say we have an image depicting a person and the features at one position of the feature map indicate a high possibiliy for the person. Now, if the corresponding receptive field contains only the body parts, the regressor has to estimate a box enclosing the entire person, although it "sees" only the body part.

Therefore, RPN creates a rough estimate of the bounding box. The second stage of Faster RCNN uses all features contained in the predicted bounding box and can correct the estimate.

In the example, RPN creates a too large bounding box, which is enclosing the person (since it cannot the see the pose of the person), and the second stage uses all information of this box to reshape it such that it is tight. This however can be done much more accurate, since more content of the object is accessable for the network.

Answer 2

According to my understanding, RPN is just for binary checking if you have Objects in the bbox or not and final Detector part is for classifying the classes ex) car, human, phones, etc

Answer 3

Short answer: no they are not redundant. The R-CNN article and its variants popularized the use of what we used to call a cascade. Back then for detection it was fairly common to use different detectors often very similar in structures to do detection because of their complementary power.

If the detections are partly orthogonal it allows to remove false positive along the way.

Furthermore by definition both parts of R-CNN have different roles the first one is used to discriminate objects from background and the second one to discriminate fine grained categories of objects from themselves (and from the background also).

But you are right if there is only 1 class vs the background one could use only the RPN part to to detection but even in that case it would probably better the result to chain two different classifiers (or not see e.g. this article)

PS: I answered because I wanted to but this question is definitely unsuited for stackoverflow

Answer 4

faster-rcnn is a two-stage method comparing to one stage method like yolo, ssd, the reason faster-rcnn is accurate is because of its two stage architecture where the RPN is the first stage for proposal generation and the second classification and localisation stage learn more precise results based on the coarse grained result from RPN.

So yes, you can, but your performance is not good enough