Storing data so that it is deleted when the program is

Question

I originally thought that the best way to save data (user settings, high scores, etc) for my application would be by writing to an embedded file in the .exe. After some research I've found that this isn't possible and that I'm better off writing to the App Data folder which is designed for this purpose.

However, I don't like this as it means that the data will persist after the .exe is deleted. It certainly isn't ideal for me as I often make many versions of apps which I ask others to test out — this would mean that their App Data folder would be cluttered with useless data from testing.

Is there a better place or way to store data which will ensure that everything belonging to the application is deleted when the .exe is?

Note that I'd like to be able to store large files, so a string in My.Settings might not be able to handle it. I'd also prefer not to write to the same directory that the program is in.