Vurforia marker tracking at the wrong position after few uses in Hololens 2

Question

I have hololens 2 (using unity for app building) and use it for tracking a marker and get the position in the hololens but after few times or few app restarting, vuforia still tracks the marker but the position it gives is wrong.

I am currently trying to find a way to place the marker position manually but I am afraid I am not succeed at all. I am pretty confused what is causing the issue here. Is hololens changing it's world origin point after some uses or vuforia is sending wrong information. I want the marker position to stay right on where marker get's tracked.