How to track multiple static points using opencv (AR)?

Question

There is an app called scandit. It’s used mainly for scanning qr codes. After the scan (multiple codes can be scanned) it starts to track them. It tracks codes based on background (AR-like). We can see it in the gif: even when I removed qr code, the point is still tracked. Tracking demonstration Can somebody recommend me a good method for making such an AR tracking?

I wanted to make the same thing. So what I have done for this:

• I am getting good points to track and their descriptors using ORB.
• Getting transformation matrix between previous and current frame using descriptors of these two frames.
• Applying transformation to the list of points that are being tracked. But there is a problem: when i move the camera too fast, the markers (of the tracking points) are losing their true positions.

So I desided to compare the descriptors of the current frame to the descriptors of the first frame. So markers are not losing their positions anymore, when I stop moving the camera even after the large move. But this approach has also a couple of problems:

• When I move the camera it can't match features properly and points are drifting while I move the camera (but when the camera stops moving, points are getting back on proper tracking)
• When the features of the first frame are not visible in the current frame, tracking is broken.

So I can’t get the solution.