VSCode python (remote) - slow "rename symbol" when opening parent/root directories?

Question

• Given I have stored my git repos at ~/git/project1, ~/git/groupA/foo, etc.
• When I open the root directory ~/git with VSCode and refactor code via "rename symbol" or "rename file",
• Then it often seems to take a long time (~30s)

My impression is that VSCode scans the whole ~/git tree, even though the refactorings definitely only affect one git repo. Is this correct? Are there tips to speed up the refactorings?