How to set multiple layers using SetWindowLong and SetLayeredWindowAttributes in C#

Question

I want to set separate layers for the Form and picBox using SetWindowLong and SetLayeredWindowAttributes in C#. But it doesn't work.

Code:

const int GWL_EXSTYLE = -20;
const uint WS_EX_LAYERED = 0x80000;
const uint LWA_ALPHA = 0x2;

void Form_Load(object? sender, EventArgs e)
{
    // Form
    SetWindowLong(Handle, GWL_EXSTYLE, WS_EX_LAYERED);
    SetLayeredWindowAttributes(Handle, 0x000000, 0, LWA_ALPHA);

    // PictureBox
    SetWindowLong(picBox1.Handle, GWL_EXSTYLE, WS_EX_LAYERED);
    SetLayeredWindowAttributes(picBox1.Handle, 0x000000, 128, LWA_ALPHA);
}

I thought this code would make the form completely transparent and the pictureBox translucent, but it ended up making the whole thing transparent.

Answer 1

You have two main managed methods to make a Layered Form:

1. Set the AllowTransparency property to true and the TransparencyKey to the BackColor value.
2. Override CreateParams to add WS_EX_LAYERED to the extended styles

Of course, you can also go unmanaged and call SetLayeredWindowAttributes.
In this case, you have to specify both LWA_ALPHA and LWA_COLORKEY as the dwFlags parameter. The effect is the same as managed method (1), given that this is what happens under the hood when you set those properties.

Since you want to have a layered Form that is completely transparent, while a Control of this Form, which should acquire the Layered status, needs to handle its own transparency, based on a different Alpha channel, I suggest you go for managed method (2).

Anyway, I'm going to show the result using both methods, since there's a not-so-subtle difference in the visual style that may be interesting.

Case 1 - Use AllowTransparency:

Set AllowTransparency and TransparencyKey in the Constructor of a Form.

public partial class SomeForm : Form {

    public SomeForm() { 
        InitializeComponent();
        AllowTransparency = true;
        TransparencyKey = BackColor;
    }
}

Case 2 - Override CreateParams:

public partial class SomeForm : Form {

    public SomeForm() => InitializeComponent();

    protected override CreateParams CreateParams {
        get { 
            var cp = base.CreateParams;
            if (OSFeature.Feature.IsPresent(OSFeature.LayeredWindows)) {
                cp.ExStyle |= (int)WS_EX_LAYERED;
            }
            return cp;
        }
    }
}

To change the layered style of a Control, you need to call the Win32 functions directly:

private void SetLayeredControl(Control control, byte alpha) {
    if (!OSFeature.Feature.IsPresent(OSFeature.LayeredWindows)) return;

    SetWindowLongPtr(control.Handle, GWL_Flags.GWL_EXSTYLE, (UIntPtr)WS_EX_LAYERED);

    SetLayeredWindowAttributes(control.Handle, 
        COLORREF.FromColor(control.BackColor), 
        alpha, 
        LayeredWindowAttr.LWA_ALPHA);
}

The relevant differences using method (1): the Caption of the Form is visible. Of course you can also set FormBorderStyle = FormBorderStyle.None to hide it. Another is that the Form is click-through, so are its layered child Controls; using method (2), the Form and its child Controls are solid.
In method (1), changing the Opacity property, sets the transparency level of both the Form and its child Controls.

In both cases, you can control the transparency level of the Layered child Control(s), resetting the bAlpha value.

A visual example:

Full code, related to just method (2).

public partial class SomeForm : Form {

    public SomeForm() => InitializeComponent();

    protected override CreateParams CreateParams {
        get { 
            var cp = base.CreateParams;
            if (OSFeature.Feature.IsPresent(OSFeature.LayeredWindows)) {
                cp.ExStyle |= (int)WS_EX_LAYERED;
            }
            return cp;
        }
    }

    protected override void OnHandleCreated(EventArgs e) {
        base.OnHandleCreated(e);
        SetLayeredControl(pictureBox1, 128);
    }

    private void SetLayeredControl(Control control, byte alpha) {
        if (!OSFeature.Feature.IsPresent(OSFeature.LayeredWindows)) return;
        SetWindowLongPtr(control.Handle, GWL_Flags.GWL_EXSTYLE, (UIntPtr)WS_EX_LAYERED);

        SetLayeredWindowAttributes(control.Handle, 
            COLORREF.FromColor(control.BackColor), 
            alpha, 
            LayeredWindowAttr.LWA_ALPHA);
    }

    [StructLayout(LayoutKind.Sequential)]
    internal struct COLORREF {
        public byte R;
        public byte G;
        public byte B;

        public COLORREF(byte red, byte green, byte blue) {
            R = red; G = green; B = blue;
        }

        public static COLORREF FromColor(Color color) => new(color.R, color.G, color.B);
    }

    internal enum GWL_Flags : int {
        GWL_EXSTYLE = -20,
        GWLP_HINSTANCE = -6,
        GWLP_HWNDPARENT = -8,
        GWL_ID = -12,
        GWL_STYLE = -16,
        GWL_USERDATA = -21,
        GWL_WNDPROC = -4,
        DWLP_USER = 0x8,
        DWLP_MSGRESULT = 0x0,
        DWLP_DLGPROC = 0x4
    }

    internal enum LayeredWindowAttr : uint {
        LWA_ALPHA = 0x00000002,     //Use bAlpha to determine the opacity of the layered window.
        LWA_COLORKEY = 0x00000001   //Use crKey as the transparency color.
    }

    const uint WS_EX_LAYERED = 0x00080000;

    [DllImport("user32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
    internal static extern UIntPtr SetWindowLongPtr(IntPtr hWnd, GWL_Flags nIndex, UIntPtr dwNewLong);

    //bAlpha = 0 -> Transparent, bAlpha = 255 -> Opaque
    [DllImport("user32.dll", SetLastError = true)]
    internal static extern bool SetLayeredWindowAttributes(IntPtr hwnd, COLORREF crKey, byte bAlpha, LayeredWindowAttr dwFlags);
}

Answer 2

No, Layered Windows doesn't work like that. WX_EX_LAYERED doesn't separate things. What it only does is make the window be processed exclusively to be blended with other windows' content. The Form is parent window for the child window Picturebox. When you make the parent transparent, you make all the contents belong to it.

The only feasible solution I can think of is to make the Form a layered window, but render the contents by using the UpdateLayeredWindow function. Before rendering the Form contents directly, generate an alpha mask from the position of the Picturebox and apply it to the bitmap that contains the contents of the Form to get the desired transcluency.

I don't explain the UpdateLayeredWindow function since there are tons of example on the net.

You can get the contents of the Form by using the Control.DrawToBitmap method.