D3DERR_INVALIDCALL in d3d->CreateDevice causing window flicker on startup in Allegro 5 D3D program

Question

I'm working on debugging window creation flicker when creating an Allegro 5 Direct3D window with multi sampling enabled. I've narrowed the problem down to window creation in allegro's d3d_disp.cpp source file. However, I can't get any debug output from DirectX. The flickering only happens in D3D mode (not OpenGL) and only when multi sampling is enabled. Also to note, this only happens when running the program on NVIDIA gpus, not on my integrated Intel.

I'm running Windows 10.

I've tried debugging this in Visual Studio 2017, but it doesn't capture debug output from DX. I installed the DirectX debug symbols when installing the DirectX SDK from June 2010.

I've tried rebuilding allegro and linking to libd3dx9d.a in gcc, but I still can't step into directx function calls and the symbols aren't loaded. There's no libd3d9d.a (note the d for debugging) library available, in MinGW-W64 GCC 8.1 or in the DirectX SDK from June 2010.

I tried running my program through PIX but it gives me an incompatibility error that I can't solve.

Here is testable example allegro 5 code :

#include <allegro5/allegro.h>
#include <allegro5/allegro_color.h>
#include <allegro5/allegro_primitives.h> 
#include <allegro5/allegro_direct3d.h>

#include <cstdio>
#include <climits>

int main(int argc, char **argv) {

    if (!al_init()) { return 1; }
    al_init_primitives_addon();
    al_install_keyboard();

    ALLEGRO_EVENT_QUEUE* queue = al_create_event_queue();
    if (!queue) { return 2; }
    al_register_event_source(queue, al_get_keyboard_event_source());

    al_set_new_display_option(ALLEGRO_SAMPLE_BUFFERS, 1, ALLEGRO_REQUIRE);
    al_set_new_display_option(ALLEGRO_SAMPLES, 8, ALLEGRO_SUGGEST);

    bool use_opengl = false;
    if (use_opengl) {
        al_set_new_display_flags(ALLEGRO_OPENGL);
    }
    else {
        al_set_new_display_flags(ALLEGRO_DIRECT3D);
    }
    ALLEGRO_DISPLAY *display = al_create_display(1024, 600);
    if (!display) { return 2; }
    if (use_opengl) {
        al_set_window_title(display, "OpenGL window");
    }
    else {
        al_set_window_title(display, "Direct3D window");
    }
    al_register_event_source(queue, al_get_display_event_source(display));


    al_clear_to_color(al_color_name("black"));
    al_draw_circle(500, 300, 200, al_color_name("white"), 5.0);
    al_draw_line(200, 200, 700, 300, al_color_name("white"), 5.0);
    al_flip_display();

    bool quit = false;
    while (!quit) {
        ALLEGRO_EVENT ev;
        al_wait_for_event(queue, &ev);
        if (ev.type == ALLEGRO_EVENT_KEY_DOWN && ev.keyboard.keycode == ALLEGRO_KEY_ESCAPE) { quit = true; }
        if (ev.type == ALLEGRO_EVENT_DISPLAY_CLOSE) { quit = true; }
    }
    return 0;
}

Ideally, the window doesn't flicker upon creation. There are many many programs that use Direct3D that don't flicker on window creation.

I've narrowed down the problem to failed calls to d3d->CreateDevice returning D3DERR_INVALIDCALL on these lines in src\win\d3d_disp.cpp in allegro's source code here : https://github.com/liballeg/allegro5/blob/master/src/win/d3d_disp.cpp#L812-L837

I need help getting the debug output from DirectX, and nothing works so far. Any tips on debugging with DirectX9, VS 2017 (and/or) MinGW-W64 GCC 8.1 and GDB, or other methods is appreciated.

EDIT An update on the things I've tried.

Defining D3D_DEBUG_INFO before including d3d9.h didn't seem to do anything when rebuilding allegro.

Enabling DirectX debugging output in the dxcpl did nothing.

Trying to run my app through PIX results in an incompatibility error. It says the directx subversions don't match between the app and the pix runtime. How do I build for a specific version of the DirectX dll?

I found that when multisampling is enabled through the D3D_PRESENT_PARAMETERS, the swap effect must be D3DSWAPEFFECT_DISCARD. Fixed that, nothing changed.

Still getting D3DERR_INVALIDCALL. I can't see anything in the presentation parameters that isn't initialized.

If I can't enable DirectX debug output I really can't tell why this error is occurring.

Debugging tips welcome. I can see that window creation fails multiple times before it succeeds and that is why the window flickers.

EDIT2 It seems to be a problem with the BackBufferFormat specified, as that is the only difference between successful window creation and failure.

EDIT3 The BackBufferFormat is fine. The real difference was the quality level of multi sampling being attempted. As per

https://github.com/liballeg/allegro5/blob/master/src/win/d3d_display_formats.cpp#L95

and

CheckDeviceMultiSampleType

there was an off by one error that made it attempt to set a quality level that was off by one. quality levels indicates the count, not the maximum index.

The flicker is gone but more testing needs to be done.

As for the supplementary question, how can I enable debug info with DirectX? Nothing I've done has worked as per the above. I will award the answer to anyone who can help me achieve debug output from D3D and DX.

@Gull_Code If you like, you can clone my testing fork of allegro here :

https://github.com/EdgarReynaldo/allegro5/tree/test

Bugsquasher

Answer 1

Each search I made on the topic returned the same 3 solutions:
-Bad driver, update or a downgrade
-Bad directx install, in the microsoft forums they asked the guy to uninstall directx, delete the root registry entry for it, reinstall directx
-A lot were pointing that it happens when the D3D struct is partially initialized without giving a fully initialized one...

I'll come back if I have some more informations.