Difference between &IntPtr and .ToPointer() in an unsafe context - P/Invoke

Question

I'm working with cuDNN and using C# with P/Invoke to wrap CUDA/CuDNN functions, which requires handling pointers in an unsafe context. Specifically, I'm implementing the function to set attributes of cuDNN backend descriptors:

[LibraryImport("cudnn64_9.dll", EntryPoint = "cudnnBackendSetAttribute")]
private unsafe static partial int CudnnBackendSetAttribute(IntPtr descriptor, int attributeName, int attributeType, long elementCount, void* arrayOfElements);
public unsafe static void SetAttribute(IntPtr descriptor, CuDnnBackendAttribute attributeName, CuDnnBackendAttributeTyp attributeType, long elementCount, void* value)
{
    CudaErrorHandler.ExecuteWithCudnnErrorHandling(() => CudnnBackendSetAttribute(descriptor, (int)attributeName, (int)attributeType, elementCount, value));
}

In some of my code, I need to pass an IntPtr to the SetAttribute function as a Descriptor_t pointer. I noticed that using &IntPtr works perfectly fine, but trying to use .ToPointer() results in a runtime exception:

// Works fine
CudaDNN.SetAttribute(fpropDesc, CuDnnBackendAttribute.CUDNN_ATTR_OPERATION_CONVOLUTION_FORWARD_X, CuDnnBackendAttributeTyp.CUDNN_TYPE_BACKEND_DESCRIPTOR, 1, &xDesc);

// Causes runtime error
CudaDNN.SetAttribute(fpropDesc, CuDnnBackendAttribute.CUDNN_ATTR_OPERATION_CONVOLUTION_FORWARD_X, CuDnnBackendAttributeTyp.CUDNN_TYPE_BACKEND_DESCRIPTOR, 1, xDesc.ToPointer());

The error I get when using .ToPointer() is:

System.Runtime.InteropServices.SEHException: "External component has thrown an exception."

Am I getting this right:

• &IntPtr gives me the address of the IntPtr itself in managed memory.
• .ToPointer() provides the pointer that IntPtr wraps, pointing to the actual resource in unmanaged memory.

If I'm right, that using &IntPtr means I'm passing the address of a managed object, then I don't understand why the cuDNN library seems to handle it correctly and interprets it without problems. Using .ToPointer(), on the other hand, results in the exception shown above.

My Questions:

• Why does using &IntPtr work here, while .ToPointer() does not?
• Could it be related to how the P/Invoke marshaller or the CLR handles IntPtr specifically?
• Is there some kind of hidden mechanism that makes &IntPtr suitable for passing to unmanaged code, despite it being a managed address?
• Is there any best practice or deeper insight into why IntPtr.ToPointer() might cause such an issue, especially when working in an unsafe context with external libraries like cuDNN?

I am specifically interested in understanding why &IntPtr works fine despite pointing to managed memory, and how the CLR ensures the right behavior when interfacing with the unmanaged backend.

Thanks in advance!