Stepping through a TCP/IP stack

Question

I was working as a QA engineer for a proprietary embedded operating system. They built their own ATN stack and stepping though it with a debugger was the most eye opening experience I have had with networking. Watching each layer of the stack build their part of the packet was amazing. Then finally being able to see the built packet on the wire had more meaning.

As an educator I would like share this experience with others. Does anyone know of a straight forward method stepping though a TCP/IP stack? Ideally I would like something easier than debugging a *BSD or Linux kernel, although if this is the only option then some tips and tricks for this process would be nice. A reference stack written in C/C++ that could be run in user mode with Visual Studio or Eclipse would be ideal.

Answer 1

You can run the NetBSD IP stack in userspace in Linux or other OS, with gdb or whatever see http://www.netbsd.org/docs/rump/ and https://github.com/anttikantee/buildrump.sh and then eg feed it to a tun/tap device so you can see whats on the wire.

Answer 2

I work in the TCP/IP industry. In BSD and variants, the function tcp_input() is an ideal starting point to explore the innards of TCP. Setting a breakpoint on this function and stepping through it on a live system can give a lot of enlightenment. If that is hard, you can simply browse through the source to get a broad feel of it:

http://fxr.watson.org/fxr/source/netinet/tcp_input.c

It will take time, many weeks at least, to understand the big picture. Quite exhilarating. :-)

Answer 3

I actually wrote a small subset of a TCP/IP stack in a 8051 once, it was a very enlightening experience.

I believe that the best way to learn something is by doing it. Once you finished your task, go and get feedback with other developers and compare your implementation with other existing ones.

My opinion might be biased here, but I think that doing this in a embedded platform is the best way to go. What you are trying to do is very low level, and a PC will just add more complexity into the problem. A embedded chip has no operational system to get in your way. Besides that, it is very satisfying to see a simple 8051 respond to ping requests and telnet calls.

They key is to start small, don't try to create a full TCP/IP stack all at once. Write the code to handle the MAC first, then IP, Ping, UDP and finally TCP.

I don't think that studying an existing implementation is a good ideia. TCP/IP implementations tend to be bloated with code that is unrelated with your goal.

Answer 4

This all depends on what you want to focus on. From your question, the thing you are most interested in is the data flow throughout the different layers (user-space stream -> voltage on the cable).

For this, I propose you use http://www.csse.uwa.edu.au/cnet/, which is a full network simulator. It allows you to step through all levels of the stack.

Real systems will always have a clear distinction between Layer3, Layer2 and Layer1 (Ethernet and CRC-checking firmware on chip, hardware MAC). You will have trouble getting into the OS and some implementation details will be messy and confusing for students. For Linux, you'll have to explain kernel infrastructure to make sense of the TCP/IP stack design.

If you are only interested in the TCP/IP part, I recommend you use an embedded TCP/IP stack like http://www.sics.se/~adam/lwip/ . You can incorporate this into a simple user-space program and fully construct the TCP/IP packet.

Please note that there are a lot of network communication aspects that you cannot address while stepping through the TCP/IP stack. There is still a MAC chip in between which regulates medium access, collisions etc. Below that, there is a PHY chip which translates everything into electric/optical signals, and there is even a protocol which handles communication between MAC and PHY. Also, you are not seeing all aspects related to queueing, concurrency, OS resource allocation ea. A full picture should include all of these aspects, which can only be seen in a network simulator.

Answer 5

Probably lwIP project is what you are looking for because it can be run without an operating system.

As for debugging Linux kernel, there is not very simple, but well-known way to do it. Use KGDB. Install debugging version of Linux kernel on virtual machine or on separate box. And remotely connect GDB to this machine. Probably you would like to use some GDB frontend instead of text-only interface. If you need more details on kernel debugging from more competent people, just add "linux" tag to the question.

Answer 6

This not exactly what you are looking for but I hope this helps

1995 - TCP/IP Illustrated, Volume 2: The Implementation (with Gary R. Wright) - ISBN 0-201-63354-X

Just walk through the code side by side. Near stepping through experience. Mr Steven's explains key variables too. Just awesome. Note: Code may have changed since the book but still awesome.

Answer 7

I would run Minix in a virtual machine and debug that. It is perfect for this.

Minix is a full OS with TCP/IP stack so you have the code you need. However, unlike Linux/BSD its roots and design goal are to be a teaching tool, so it eschews a certain level of complexity in favor of being clear. In fact, this is the OS Linus Torvalds started hacking on when he started out with Linux :-)

You can run minix in an VM such as VirtualBox or VMware and debug it. There are instruction on the web site: http://www.minix3.org/

Answer 8

I personally learned TCP/IP stack using DOS and SoftICE (oops, leaked that I'm an old guy). Using DOS on a virtual machine and debug through a TCP/IP driver will be much simpler since your goal is to educate how TCP/IP works. Modern OS does a lot of optimization on network I/O and it's not easy to debug through.

http://www.crynwr.com/ has a bunch of open source packet drivers. Debugging with source code shall be a bit easier.