Latency in video streaming from backend to frontend (and vice versa)

Question

I am building a mobile app in flutter X firebase, with the backend in Django. The issue is, my Computer vision algorithm is on the front end, and it sends live key points to the backend where the classifier classifies, and sends the result back to the frontend. This is causing a huge delay of 200ms+. How can I fix this?

I tried the CV algo on the frontend, and classifier on the backend to reduce the complexity of the server side. But im out of ideas now. Please let me know if establishing a websocket would help me achieve the desired goals. Also, I wanted to know if i could host the algo on a server rather then using the client's mobile's computation power to reduce the latency. Please note that i am in dire of reducing the latency. All answers are appreciated, thanks!