How do commands and programs communicate with the Window OS

Question

How do programs made for windows interact with, or issue commands to, the kernel of Windows NT?

And how does the kernel return any data?

Answer 1

In order to answer this question, it's important to understand the difference between user and kernel mode. Kernel mode is the most privileged CPU mode, where executing code has complete access to the hardware. It is used for the most low-level operating system functionality. User mode is a much more restricted CPU mode. It prevents code from directly accessing the hardware. Applications run in user mode. Of course, they still need to access the hardware somehow, so they need to call into the kernel.

That's where your question leads to. In order to allow user mode code to call into the kernel, the Windows kernel sets up an entry point. On x86-based systems, this entry point is either a software interrupt (int 2e) or the sysenter/syscall instruction. Executing these instructions causes a CPU mode switch, transitioning the CPU from user mode into kernel mode. Once the CPU has switched modes, it calls a function specified by the kernel. In Windows, this function is the system service dispatcher.

The system service dispatcher is responsible for calling the kernel service that the user mode code wants. It takes the function number, which was specified by the user mode code, and looks it up in the system service descriptor table (SSDT). The SSDT is basically a list of function pointers to each kernel services. Once it identifies the correct kernel service, it calls it with the arguments that the user mode application also specified. After the kernel service completes, the CPU returns to the application, either through the iret instruction (if coming from a software interrupt) or sysexit/sysret (if coming from sysenter/syscall).

All of this sounds quite complex, and it is, which is why Windows hides these details from programmers. Instead of requiring them to directly communicate with the kernel through the entry point the kernel sets up, Windows provides programmers with several DLLs, which do this for them.

Now here's where it once again gets somewhat more complicated. The process of calling kernel services from user mode is implemented in ntdll.dll, but ntdll.dll isn't directly used by most programmers. Instead, it exports a generic set of kernel services called the Native API. Above this, the Win32 API is implemented in kernel32.dll. Most functions in kernel32.dll are simply wrappers of functions in ntdll.dll.

You might ask why this is done. Why not just have kernel32.dll call the kernel functions directly? The reason for this is to allow for different multiple user mode APIs. Windows NT was designed to support multiple APIs, not just Win32, but also POSIX and OS/2. Each user mode API calls into ntdll.dll to implement their own APIs, preventing them from needing to directly call kernel services themselves.

Answer 2

Try reading this: Chapter 5 - Windows NT 4.0 Workstation Architecture. It should be enough to start.

In the end some API are built directly in some userland DLL. These are executed directly. Other require kernel-mode help/services.

For these (an I'm quoting from the link above)

The application uses an API to make a graphics call to a user-mode dynamic link library. The component that implements the call makes a kernel-mode trap call to the Executive to switch its thread and copy its call parameters from its user mode stack to its kernel mode stack. Then, the processor's stack register is switched to point to kernel mode. Now the thread can run in the Executive.

Applications communicate with protected subsystems by using local procedure calls (LPC), an application-independent method of communication between components on the same computer. After the thread is switched to kernel mode, the Microkernel schedules the LPC for delivery.

Using Fast LPC, an optimized communication method, the Microkernel recognizes that the call from the application involves a thread in the environment subsystem. It considers the calling application thread and the receiving subsystem thread to be paired. The receiving thread can now use the unexpired time from the sending thread.

The graphics call parameters are passed to the receiving subsystem thread. The receiving thread switches back to user mode to fulfill the graphics request.

The subsystem completes its task and then returns control to the waiting calling thread in the application by the same method.

The calling thread (from the DLL) is switched back to user mode before returning control to the application.

Microsoft operating-system engineers also used the concept of a shared memory window to speed up communication. Data is placed in a temporary shared memory window administered by the Process Manager in the Executive. This lets the application see into the subsystem's memory and share data without using LPCs. However, because the application thread must still run in the Executive, kernel/user mode transitions and thread switches are still required.

Here there are some notes on how exactly is the call done (what is the ASM command used): The system call dispatcher on x86 if you need it.

Answer 3

Dude, that's a very broad question.

I recommend the book Windows Internals By Mark Russinovich et ag if you really really want to under stand this. Another good book is the classic Operating Systems by Deitel et al.

Start however with Inside Windows NT by Helen Custer (edition 1) - it's a very basic book (note that last link has a pic of the coverof edition 2 - which is way way way more detailed).

Ok in a nutshell.

There are a variety of protocols for communcation between windows components. Most of them will employ passing data via some shared memory (such as buffers, stack etc) at the end of the day. But the protocols can be very involved and are different for different communications.

My suggestion to you is have a loock at the above books and determine how the architecture of the Windows operating system hangs together. From here you'll see how the various components communicate.

(applying nerd face) - Trust me those are great books for learning about Windows and operating systems in general if that's what floats your boat.