Slow breakpoint triggering with GDB and LinkServer/MCULink

Question

I'm working on a project involving the NXP i.MX RT1060, and I'm using NXP's MCULink for debugging. I use NXP's LinkServer for running a GDB server, and the Cortex-Debug extension in VSCode for GDB. Finally, the project is running using the ThreadX RTOS. Here's versioning information:

Ubuntu 22.04 (x86_64)
VSCode v1.91.1
cortex-debug v1.12.1
GDB multiarch v12.1
LinkServer v1.5.30
MCU-LINK (r0FF) CMSIS-DAP V3.140

Now, the issue I'm experiencing is that when there are 15 or more threads alive in the system when a breakpoint is hit, it takes a long time (> 30s) to bring me to the breakpoint and allowing me to interact with the debugger. If there are 14 or less it happens "instantaneously". The things I've tried are as follows:

• I've gone through the output logs from GDB and see that there are packet error messages when its slow (Ignoring packet error, continuing...), but not when it's fast. These appear after thread-info N is called on the various threads in the system
• I've set set remotetimeout 1 (default is 2), and this speeds things up when it's slow, but it's still slow
• I've set set debug remote 1, which gives me the following output:

0000018993+00000: -> ~"Thread 7 hit Breakpoint 2, app () at /workspaces/connected2/app/src/app.cpp:652\n"
0000018993+00000: Thread 7 hit Breakpoint 2, app () at /workspaces/connected2/app/src/app.cpp:652
0000018993+00000: -> ~"652\t            auto message = application_queue->receive(Wait::DontWait());\n"
0000018993+00000: 652               auto message = application_queue->receive(Wait::DontWait());
0000018993+00000: -> *stopped,reason="breakpoint-hit",disp="keep",bkptno="2",frame={addr="0x6011da94",func="app",args=[],file="/workspaces/connected2/app/src/app.cpp",fullname="/workspaces/connected2/app/src/app.cpp",line="652",arch="armv7e-m"},thread-id="7",stopped-threads="all"
0000018993+00000: mi2.status = stopped
0000018994+00001: 31-thread-list-ids
0000018994+00000: -> &"[remote] Sending packet: $qfThreadInfo#bb\n"
0000018994+00000: [remote] Sending packet: $qfThreadInfo#bb
0000019002+00008: -> &"[remote] Received Ack\n"
0000019002+00000: [remote] Received Ack
0000019002+00000: -> &"[remote] Packet received: m20015AB8,20002230,20002410,20002500,20002320,200028C0,200027D0,200026E0,200025F0,2003A810,20039158,2003B5F0,200029B0,20002AA0,20002B90,20002C80,20002D70,20002E60,20002F50\n"
0000019002+00000: [remote] Packet received: m20015AB8,20002230,20002410,20002500,20002320,200028C0,200027D0,200026E0,200025F0,2003A810,20039158,2003B5F0,200029B0,20002AA0,20002B90,20002C80,20002D70,20002E60,20002F50
0000019002+00000: -> &"[remote] Sending packet: $qsThreadInfo#c8\n"
0000019002+00000: [remote] Sending packet: $qsThreadInfo#c8
0000019002+00000: -> &"[remote] Received Ack\n"
0000019002+00000: [remote] Received Ack
0000019002+00000: -> &"[remote] Packet received: lOK\n"
0000019002+00000: [remote] Packet received: lOK
0000019002+00000: -> 31^done,thread-ids={thread-id="2",thread-id="3",thread-id="4",thread-id="5",thread-id="6",thread-id="7",thread-id="8",thread-id="9",thread-id="10",thread-id="11",thread-id="12",thread-id="13",thread-id="14",thread-id="15",thread-id="16",thread-id="17",thread-id="18",thread-id="19",thread-id="20"},current-thread-id="7",number-of-threads="19"
0000019002+00000: 32-thread-info 2
0000019002+00000: -> &"[remote] Sending packet: $qfThreadInfo#bb\n"
0000019002+00000: [remote] Sending packet: $qfThreadInfo#bb
0000019008+00006: -> &"[remote] Received Ack\n"
0000019008+00000: [remote] Received Ack
0000019008+00000: -> &"[remote] Packet received: m20015AB8,20002230,20002410,20002500,20002320,200028C0,200027D0,200026E0,200025F0,2003A810,20039158,2003B5F0,200029B0,20002AA0,20002B90,20002C80,20002D70,20002E60,20002F50\n"
0000019008+00000: [remote] Packet received: m20015AB8,20002230,20002410,20002500,20002320,200028C0,200027D0,200026E0,200025F0,2003A810,20039158,2003B5F0,200029B0,20002AA0,20002B90,20002C80,20002D70,20002E60,20002F50
0000019008+00000: -> &"[remote] Sending packet: $qsThreadInfo#c8\n"
0000019008+00000: [remote] Sending packet: $qsThreadInfo#c8
0000019008+00000: -> &"[remote] Received Ack\n"
0000019008+00000: [remote] Received Ack
0000019008+00000: -> &"[remote] Packet received: lOK\n"
0000019008+00000: [remote] Packet received: lOK
0000019008+00000: -> &"[remote] Sending packet: $qThreadExtraInfo,20002d70#44\n"
0000019008+00000: [remote] Sending packet: $qThreadExtraInfo,20002d70#44
0000019020+00012: -> &"[remote] Received Ack\n"
0000019020+00000: [remote] Received Ack
0000019020+00000: -> &"[remote] Packet received: 226d6f64756c65732f6d6963726f6f63707022203a2054585f534c454550\n"
0000019020+00000: [remote] Packet received: 226d6f64756c65732f6d6963726f6f63707022203a2054585f534c454550
0000019020+00000: -> &"[remote] Sending packet: $Hg20002d70#6e\n"
0000019020+00000: [remote] Sending packet: $Hg20002d70#6e
0000019020+00000: -> &"[remote] Received Ack\n"
0000019020+00000: [remote] Received Ack
0000019020+00000: -> &"[remote] Packet received: OK\n"
0000019020+00000: [remote] Packet received: OK
0000019020+00000: -> &"[remote] Sending packet: $g#67\n"
0000019020+00000: [remote] Sending packet: $g#67
0000019026+00006: -> &"[remote] Received Ack\n"
0000019026+00000: [remote] Received Ack
0000019026+00000: -> &"[remote] read_frame: Bad checksum, sentsum=0x25, csum=0xe2, buf=000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000ce924200000000000000000\\000002B90,20002C80,20002D70,20002E60,20002F50\n"
0000019026+00000: [remote] read_frame: Bad checksum, sentsum=0x25, csum=0xe2, buf=000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000ce924200000000000000000\000002B90,20002C80,20002D70,20002E60,20002F50
0000020027+01001: -> &"[remote] getpkt_or_notif_sane_1: Timed out.\n"
0000020027+00000: [remote] getpkt_or_notif_sane_1: Timed out.
0000021028+01001: -> &"[remote] getpkt_or_notif_sane_1: Timed out.\n"
0000021028+00000: [remote] getpkt_or_notif_sane_1: Timed out.
0000021028+00000: -> ~"Ignoring packet error, continuing...\n"
0000021028+00000: Ignoring packet error, continuing...
0000021029+00001: -> 32^done,threads=[{id="2",target-id="Thread 536882544",details="\"modules/microocpp\" : TX_SLEEP",frame={level="0",addr="0x00000000",func="??",args=[],arch="armv7e-m"},state="stopped"}]

As the log shows, everything seems fine until read_frame gets a bad checksum, and getpkt_or_notif_sane_1 times out. However, even though something wrong happens the actual thread information is retrieved OK, and the debugging seems to work fine, but each step-into/over and continue is just as slow.

I've tried reaching out to NXP, but haven't heard back yet, so I'm wondering if anyone knows what could be causing this, and if there's a way to fix it?

Answer 1

The slow behaviour you see is 100% caused by the checksum error, which I'm almost certain is caused by either the remote target sending the wrong checksum, or corruption "on the wire" between the remote target and GDB.

I say this with confidence because the checksum calculation code in GDB hasn't changed in many years, and I've never seen a checksum issue that wasn't caused by the remote target.

When GDB gets a checksum error it sends a NAK packet (-) back to the remote and waits for a resend, this behaviour is documented here. GDB will then wait for the remote timeout, send another NAK packet, and then wait again. GDB will make 3 attempts to fetch the packet and then give up.

Which lines up with your debug trace, the first attempt does result in a packet, but the checksum is wrong. The next two attempts both time out, then GDB moves on.

From the evidence you've provided I would suggest that in this case GDB is behaving as expected and the remote target is broken with the following two issues:

1. Sending the wrong checksum in response to a g packet, and
2. Not handling a NAK reply from GDB -- though I suspect most targets probably assume the connection is reliable so don't handle NAK correctly. Still, it would be nice if the target spotted the NAK from GDB, realised things have gone wrong and signalled an error.

I know this answer doesn't help solve your problem, but I thought I'd add what I could.