Salesforce issue ENOENT: no such file or directory, open '/scripts/scratchOrg/preconfiguration/auth/server.key'

Question

I have encountered a very annoying issue on VSCode for the salesforce project. I had a file called server.key but deleted it a while ago (and btw I am not using Scratch Org). This file is only in /scripts/ folder and not in salesforce org. However, a few days later, no matter whether I retrieve or deploy any file, or run any other command, it gives me an error: ENOENT: no such file or directory, open '/scripts/scratchOrg/preconfiguration/auth/server.key'. I tried to reauthorise, uninstall cli, uninstall Salesforce Extension Pack and even removed the whole project and VScode. All these only works for a while but then this error keeps appearing. I don't understand why it keeps pointing to this deleted file, even though I want to deploy/receive a totally different file