Performance-related: How does SQL Server process concurrent queries from multiple connections from a single .NET desktop application?

Question

Single-threaded version description:

1. Program gathers a list of questions.
   
2. For each question, get model answers, and run each one through a scoring module.

---

• Scoring module makes a number of (read-only) database queries.
• Serial processing, single database connection.

I decided to multi-thread the above described program by splitting the question list into chunks and creating a thread for each one.

Each thread opens it's own database connection and works on it's own list of questions (about 95 questions on each of 6 threads). The application waits for all threads to finish, then aggregates the results for display.

To my surprise, the multi-threaded version ran in approximately the same time, taking about 16 seconds instead of 17.

Questions:

Why am I not seeing the kind of gain in performance I would expect from executing queries concurrently on separate threads with separate connections? Machine has 8 processors.

Will SQL Server process queries concurrently when they are coming from a single application, or might it (or .net itself) be serializing them?

Might there be something misconfigured, that would make it go faster, or might I just be pushing SQL Server to its computational limits?

Current configuration:

Microsoft SQL Server Developer Edition 9.0.1406 RTM
OS: Windows Server 2003 Standard
Processors: 8
RAM: 4GB

Answer 1

This is just a shot in the dark, but I bet you are not seeing the performance gain because they serialize themselves in the database due to locking of shared resources (records). Now for the small print.

I assume your C# code is actually correct and you actually do start separate threads and issue each query in parallel. No offense, but I've seen many making that claim and the code being actually serial in the client, for various reasons. You should validate this by monitoring the server (via Profiler, or use the sys.dm_exec_requests and sys.dm_exec_sessions).

Also I assume that your queries are of similar weight. i.e., you do not have one thread that lasts 15 seconds and 5 that 100 ms.

The symptoms you describe, in lack of more details, would point that you have a write operation at the beginning of each thread that takes an X lock on some resource. First thread starts and locks the resource, other 5 wait. 1st thread is done, releases the resource then the next one grabs it, other 4 wait. So last thread has to wait for the execution of all other 5. This would be extremely easy to troubleshoot by looking at sys.dm_exec_requests and monitor what blocks the requests.

BTW you should consider using Asynchronous Processing=true and rely on the async methods like BeginExecuteReader to launch your commands in execution in parallel w/o the overhead of client side threads.

Answer 2

You can simply check the task manager when the process is running. If it's showing 100% CPU usage then its CPU bound. Otherwise its IO Bound.

For hyperthreading 50% CPU usage is roughly equal to 100% usage!

Wow I didn't realize how old the thread was. I guess its always good to leave the response for others looking.

Answer 3

I ran a join query across sys.dm_os_workers, sys.dm_os_tasks, and sys.dm_exec_requests on task_address, and here are the results (some uninteresting/zero-valued fields excluded, others prefixed with ex or os to resolve ambiguities):

-COL_NAME-  -Thread_1-  -Thread_2-  -Thread_3-  -Thread_4-

task_state  SUSPENDED   SUSPENDED   SUSPENDED   SUSPENDED
context_switches_count  2   2   2   2
worker_address  0x3F87A0E8  0x5993E0E8  0x496C00E8  0x366FA0E8
is_in_polling_io_completion_routine 0   0   0   0
pending_io_count    0   0   0   0
pending_io_byte_count   0   0   0   0
pending_io_byte_average 0   0   0   0
wait_started_ms_ticks   1926478171  1926478187  1926478171  1926478187
wait_resumed_ms_ticks   1926478171  1926478187  1926478171  1926478187
task_bound_ms_ticks 1926478171  1926478171  1926478156  1926478171
worker_created_ms_ticks 1926137937  1923739218  1921736640  1926137890
locale  1033    1033    1033    1033
affinity    1   4   8   32
state   SUSPENDED   SUSPENDED   SUSPENDED   SUSPENDED
start_quantum   3074730327955210    3074730349757920    3074730321989030    3074730355017750
end_quantum 3074730334339210    3074730356141920    3074730328373030    3074730361401750
quantum_used    6725    11177   11336   6284
max_quantum 4   15  5   20
boost_count 999 999 999 999
tasks_processed_count   765 1939    1424    314
os.task_address 0x006E8A78  0x00AF12E8  0x00B84C58  0x00D2CB68
memory_object_address   0x3F87A040  0x5993E040  0x496C0040  0x366FA040
thread_address  0x7FF08E38  0x7FF8CE38  0x7FF0FE38  0x7FF92E38
signal_worker_address   0x4D7DC0E8  0x571360E8  0x2F8560E8  0x4A9B40E8
scheduler_address   0x006EC040  0x00AF4040  0x00B88040  0x00E40040
os.request_id   0   0   0   0
start_time  2009-05-26 19:39    39:43.2 39:43.2 39:43.2
ex.status   suspended   suspended   suspended   suspended
command SELECT  SELECT  SELECT  SELECT
sql_handle  0x020000009355F1004BDC90A51664F9174D245A966E276C61  0x020000009355F1004D8095D234D39F77117E1BBBF8108B26  0x020000009355F100FC902C84A97133874FBE4CA6614C80E5  0x020000009355F100FC902C84A97133874FBE4CA6614C80E5
statement_start_offset  94  94  94  94
statement_end_offset    -1  -1  -1  -1
plan_handle 0x060007009355F100B821C414000000000000000000000000  0x060007009355F100B8811331000000000000000000000000  0x060007009355F100B801B259000000000000000000000000  0x060007009355F100B801B259000000000000000000000000
database_id 7   7   7   7
user_id 1   1   1   1
connection_id   BABF5455-409B-4F4C-9BA5-B53B35B11062    A2BBCACF-D227-466A-AB08-6EBB56F34FF2    D330EDFE-D49B-4148-B7C5-8D26FE276D30    649F0EC5-CB97-4B37-8D4E-85761847B403
blocking_session_id 0   0   0   0
wait_type   CXPACKET    CXPACKET    CXPACKET    CXPACKET
wait_time   46  31  46  31
ex.last_wait_type   CXPACKET    CXPACKET    CXPACKET    CXPACKET
wait_resource               
open_transaction_count  0   0   0   0
open_resultset_count    1   1   1   1
transaction_id  3052202 3052211 3052196 3052216
context_info    0x  0x  0x  0x
percent_complete    0   0   0   0
estimated_completion_time   0   0   0   0
cpu_time    0   0   0   0
total_elapsed_time  54  41  65  39
reads   0   0   0   0
writes  0   0   0   0
logical_reads   78745   123090  78672   111966
text_size   2147483647  2147483647  2147483647  2147483647
arithabort  0   0   0   0
transaction_isolation_level 2   2   2   2
lock_timeout    -1  -1  -1  -1
deadlock_priority   0   0   0   0
row_count   6   0   1   1
prev_error  0   0   0   0
nest_level  2   2   2   2
granted_query_memory    512 512 512 512

The query plan predictor for all queries shows a couple nodes, 0% for select, and 100% for a clustered index seek.

Edit: The fields and values I left out where (same for all 4 threads, except for context_switch_count): exec_context_id(0), host_address(0x00000000), status(0), is_preemptive(0), is_fiber(0), is_sick(0), is_in_cc_exception(0), is_fatal_exception(0), is_inside_catch(0), context_switch_count(3-89078), exception_num(0), exception_Severity(0), exception_address(0x00000000), return_code(0), fiber_address(NULL), language(us_english), date_format(mdy), date_first(7), quoted_identifier(1), ansi_defaults(0), ansi_warnings(1), ansi_padding(1), ansi_nulls(1), concat_null_yields_null(1), executing_managed_code(0)

Answer 4

is it possible that the the threads share a connection? did you verify that multiple SPIDs are created when this runs (sp_who)?

Answer 5

My first inclination is that you're trying to solve an IO problem with threads, which almost never works. IO is IO, and more threads doesn't increase the pipe. You'd be better off downloading all questions and their answers in one batch and processing the batch locally with multiple threads.

Having said that, you're probably experiencing some db locking that is causing slowness. Since you're talking about read-only queries, try using the with (nolock) hint on your queries to see if that helps.

Regarding SQL server processing, it is my understanding that SQL Server will try to process as many connections concurrently as possible (one statement at a time per connection), up to the max connections allowed by configuration. The kind if issue you're seeing is almost never a thread issue and almost always a locking or IO problem.

Answer 6

How large is your database? How fast are your HDDs / Raid / Other storage

Perhaps your DB is I/O bound?