Marshalling from c# LPCTSTR doesn't work any more

Question

I use P/Invoke in one of my projects to call avifil32.dll methods and everything worked fine so far. Today, being back to that project after some months and by having switched to Windows 8 in the meantime, I've discovered that nothing works anymore. Most of my calls to avifil32 methods causes an AccessViolationException ("Attempted to read or write protected memory...").

One function causing this was AVIFileOpen, which previously I was calling in this way (as also stated on pinvoke.net here):

[DllImport("avifil32.dll", PreserveSig=true)]
static extern int AVIFileOpen(out IntPtr ppfile, string szFile, uint mode, int pclsidHandler);

After some search on the web I found this useful post and consequently I changed my call to (note AVIFileOpenW):

[DllImport("avifil32.dll", PreserveSig=true, CharSet = CharSet.Auto)]
public static extern int AVIFileOpenW(ref int ppfile, [MarshalAs(UnmanagedType.LPWStr)] String szFile,  int uMode, int pclsidHandler);

and now it works. However I have other problems, for instance I've the same exception on the call to AVIFileCreateStream

So what am I doing wrong? And even more interesting, which can be the reason that causes nothing to work any more now? I admit that I do not know what could have happened in the meantime. The big change has been passing from Windows 7 to Windows 8, but could this explain the problem?

EDIT

---

Following Hans Passant sugestion I've corrected the P/Invoke call and now I've no more the AccessViolationexception. However it is still a mistery to me while before was working. In fact both now (Win8) and before (Win7) I had 64bit OS. However there is another detail in the project settings that may can help to explain. This is a library project that was called from my main project. My main project targets x64 (targeting x86 is not an option), while the library targets AnyCPU. I did another trial and again with these settings and my old version of the the AVIFileOpen call the program works under a Win7 x64 machine but not in a Win8 x64 one. Can this be related to a different "management" of the AnyCPU by the JIT compiler, or there must be some other settings that I'm missing?

Now the program works both in x86 and in x64. Here below an extract of the code I'm using:

AVIFileInit();
IntPtr aviFile = IntPtr.Zero;
IntPtr aviStream = IntPtr.Zero;
string tmp = Path.GetTempFileName();
if (AVIFileOpen(out aviFile, tmp.Substring(0, tmp.Length - 4) + ".avi", OF_WRITE | OF_CREATE, IntPtr.Zero) == 0)
{
    AVISTREAMINFO strhdr = new AVISTREAMINFO();
    strhdr.fccType = mmioStringToFOURCC("vids", 0);
    strhdr.fccHandler = mmioStringToFOURCC("CVID", 0);
    strhdr.dwScale = 1;
    strhdr.dwRate = 25;
    strhdr.dwSuggestedBufferSize = 102400;
    strhdr.dwQuality = -1;
    strhdr.rcFrame.bottom = 320;
    strhdr.rcFrame.right = 320;
    strhdr.szName = "";
    if (AVIFileCreateStream(aviFile, out aviStream, ref strhdr) == 0)
    {
        AVICOMPRESSOPTIONS_CLASS options = new AVICOMPRESSOPTIONS_CLASS();
        options.fccType = (uint)streamtypeVIDEO;
        options.lpParms = IntPtr.Zero;
        options.lpFormat = IntPtr.Zero;
        bool ok = AVISaveOptions(this.Handle, ICMF_CHOOSE_KEYFRAME | ICMF_CHOOSE_DATARATE, 1, ref aviStream, ref options);
    }
}

public static readonly int streamtypeVIDEO = mmioFOURCC('v', 'i', 'd', 's');
public const UInt32 ICMF_CHOOSE_KEYFRAME = 0x0001;
public const UInt32 ICMF_CHOOSE_DATARATE = 0x0002;
public const UInt32 ICMF_CHOOSE_PREVIEW = 0x0004;
public const int OF_WRITE = 1;
public const int OF_READWRITE = 2;
public const int OF_CREATE = 4096;

[StructLayout(LayoutKind.Sequential)]
public struct RECT
{
    public UInt32 left;
    public UInt32 top;
    public UInt32 right;
    public UInt32 bottom;
} 

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
public struct AVISTREAMINFO
{
    public Int32 fccType;
    public Int32 fccHandler;
    public Int32 dwFlags;
    public Int32 dwCaps;
    public Int16 wPriority;
    public Int16 wLanguage;
    public Int32 dwScale;
    public Int32 dwRate;
    public Int32 dwStart;
    public Int32 dwLength;
    public Int32 dwInitialFrames;
    public Int32 dwSuggestedBufferSize;
    public Int32 dwQuality;
    public Int32 dwSampleSize;
    public RECT rcFrame;
    public Int32 dwEditCount;
    public Int32 dwFormatChangeCount;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 64)] public String szName;
}

[StructLayout(LayoutKind.Sequential)]
public class AVICOMPRESSOPTIONS_CLASS
{
    public UInt32 fccType;
    public UInt32 fccHandler;
    public UInt32 dwKeyFrameEvery;
    public UInt32 dwQuality;
    public UInt32 dwBytesPerSecond;
    public UInt32 dwFlags;
    public IntPtr lpFormat;
    public UInt32 cbFormat;
    public IntPtr lpParms;
    public UInt32 cbParms;
    public UInt32 dwInterleaveEvery;
}

[DllImport("avifil32.dll")]
public static extern void AVIFileInit();

[DllImport("winmm.dll", CharSet = CharSet.Auto)]
public static extern int mmioStringToFOURCC([MarshalAs(UnmanagedType.LPWStr)] String sz, int uFlags);

[DllImport("avifil32.dll", PreserveSig = true, CharSet = CharSet.Auto)]
public static extern int AVIFileOpen(out IntPtr ppfile, String szFile, int uMode, IntPtr pclsidHandler);

[DllImport("avifil32.dll", CharSet = CharSet.Auto)]
public static extern int AVIFileCreateStream(IntPtr pfile, out IntPtr ppavi, ref AVISTREAMINFO ptr_streaminfo);

[DllImport("avifil32.dll")]
public static extern bool AVISaveOptions(IntPtr hwnd, UInt32 uiFlags, Int32 nStreams, ref IntPtr ppavi, ref AVICOMPRESSOPTIONS_CLASS plpOptions);

Answer 1

The big change has been passing from Windows 7 to Windows 8

That can explain the problems, you probably got the 64-bit version of Windows 8. Both declarations you used are wrong and will malfunction in very hard to diagnose ways when you run your program in 64-bit mode. The relevant project setting is Project + Properties, Build tab, Platform target setting. Best to use "x86" here (tick Prefer 32-bit in VS2012 and up), that will limit the misery you'll suffer from these bad declarations.

The 1st argument is PAVIFILE, "pointer to a buffer that receives the interface pointer". The last argument is CLSID, "pointer to a class identifier". Pointers are 4 bytes in 32-bit mode, 8 bytes in 64-bit mode. The corresponding C# declaration must use either the ref or out keyword or use IntPtr to be compatible.

So the 1st argument is a "pointer to a pointer", using out IntPtr is correct. Like you had.

The last argument is almost never used and you pass a null pointer. In which case you want to declare it IntPtr and pass IntPtr.Zero when you make the call. You originally passed 0, 4 bytes while the function expected 8 bytes set to 0. Enough to generate a bad pointer value and get the AccessViolationException.

The replacement declaration you used is especially nasty, you only passed enough space to store an int, 4 bytes, but the function writes 8 bytes in 64-bit mode. That will corrupt the stack or the GC heap, very nasty. A subsequent call, like AviFileCreateStream is indeed very likely to fail since the interface pointer value is not likely to survive untouched.

I fixed the pinvoke.net declarations.