How to find performance hot spots in .Net application?

Question

I have an existing ASP.NET 2.0 web service serves several WinForms clients. In our application, We belive we have performance problem in several levels.

1. Sending toomuch data in syncrounous request
2. Lazy loading, Too many round trip between web service and database
3. POCO <-> Sql object mapping using untyped datasets and reflection[no caching]

This is an existing application with large code base, I would like to instrument this app to find out hotspots.

1. How can I instrument remote apps like winforms client deployed in remote places?
2. How can I instrument Web Service?
3. Edit** Are there any profilers better than VS profiler?
4. Can I trust the profilers to tell me the hot spots, so I don't pollute my code with instrumentation? Or Do I have to take the middle road, profiler + instrumentation?

Answer 1

This is a great tool for profiling and troubleshooting a live web app: http://newrelic.com/

Answer 2

This may help:
ASP.NET and WorkerThreads – Interesting performance tweaks
20 Tips to Improve ASP.net Application Performance
Use Custom Http Handlers To Improve Performance in ASP.NET
Best practices for ASP.Net applications
Scaling Strategies for ASP.NET Applications

Answer 3

I'd recommend checking out dotTrace. I confess I haven't used this, but I've used their other flagship product (Resharper) which is outstanding.