Proper image layout transitions between two Render Passes in Vulkan

Question

I'm trying to integrate DearImGui with my Vulkan application in a way that the application and DearImGui use seperate Render Passes, but the same Command Buffer.

The first Render Pass (belonging to the application) should write into a color attachment and the second Render Pass (belonging to DearImGui) should take this color output and record its commands over it, however when I run my program in the current state I get the following errors:

validation layer \[XXXXX\]: Validation Error: \[ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout \] Object 0: handle = 0x20221446a60, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits\[0\].pCommandBuffers\[0\] command buffer VkCommandBuffer 0x20221446a60\[\] expects VkImage 0xf56c9b0000000004\[\] (subresource: aspectMask 0x1 array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL--instead, current layout is VK_IMAGE_LAYOUT_UNDEFINED.
validation layer \[XXXXX\]: Validation Error: \[ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout \] Object 0: handle = 0x202214481c0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits\[0\].pCommandBuffers\[0\] command buffer VkCommandBuffer 0x202214481c0\[\] expects VkImage 0xe7f79a0000000005\[\] (subresource: aspectMask 0x1 array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL--instead, current layout is VK_IMAGE_LAYOUT_UNDEFINED.
validation layer \[XXXXX\]: Validation Error: \[ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout \] Object 0: handle = 0x20221446a60, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits\[0\].pCommandBuffers\[0\] command buffer VkCommandBuffer 0x20221446a60\[\] expects VkImage 0xf443490000000006\[\] (subresource: aspectMask 0x1 array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL--instead, current layout is VK_IMAGE_LAYOUT_UNDEFINED.
validation layer \[XXXXX\]: Validation Error: \[ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout \] Object 0: handle = 0x202214481c0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits\[0\].pCommandBuffers\[0\] command buffer VkCommandBuffer 0x202214481c0\[\] expects VkImage 0xf56c9b0000000004\[\] (subresource: aspectMask 0x1 array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL--instead, current layout is VK_IMAGE_LAYOUT_PRESENT_SRC_KHR.
validation layer \[XXXXX\]: Validation Error: \[ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout \] Object 0: handle = 0x20221446a60, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits\[0\].pCommandBuffers\[0\] command buffer VkCommandBuffer 0x20221446a60\[\] expects VkImage 0xe7f79a0000000005\[\] (subresource: aspectMask 0x1 array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL--instead, current layout is VK_IMAGE_LAYOUT_PRESENT_SRC_KHR.
validation layer \[XXXXX\]: Validation Error: \[ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout \] Object 0: handle = 0x202214481c0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits\[0\].pCommandBuffers\[0\] command buffer VkCommandBuffer 0x202214481c0\[\] expects VkImage 0xf443490000000006\[\] (subresource: aspectMask 0x1 array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL--instead, current layout is VK_IMAGE_LAYOUT_PRESENT_SRC_KHR.
validation layer \[XXXXX\]: Validation Error: \[ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout \] Object 0: handle = 0x20221446a60, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits\[0\].pCommandBuffers\[0\] command buffer VkCommandBuffer 0x20221446a60\[\] expects VkImage 0xf56c9b0000000004\[\] (subresource: aspectMask 0x1 array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL--instead, current layout is VK_IMAGE_LAYOUT_PRESENT_SRC_KHR.
...

As I understand the Render Passes implicitly transition the layout of the images they work on after they are finished with their operations. Here however this does not happen, my best guess is that the synchronisation between the Render Passes dont work correctly. I tried setting up the subpass dependecies following this post: https://frguthmann.github.io/posts/vulkan_imgui.

I'm fairly new to Vulkan and I have been stuck on this issue for a while now. As I understand the subpass dependecies make sure that the srcSubpass finishes every operation that is the type of srcAccessMask at the srcStageMask stage before letting the dstSubpass start on the operations that are the type of dstAccessMask at the dstStageMask stage. Based on this the second subpass should wait with the rendering until the first subpass is finished writing into the color attachment, but I failed to confiure the subpass dependencies that way.

The initialization of the first Render Pass belonging to the application looks like this:

void vulkanRenderer::initRenderPass(vk::Format imageFormat) {
    auto colorAttachment = vk::AttachmentDescription(
        vk::AttachmentDescriptionFlags(),
        imageFormat,
        msaaSamples,
        vk::AttachmentLoadOp::eClear,
        vk::AttachmentStoreOp::eStore,
        vk::AttachmentLoadOp::eDontCare,
        vk::AttachmentStoreOp::eDontCare,
        vk::ImageLayout::eUndefined,
        vk::ImageLayout::eColorAttachmentOptimal
    );

    auto depthAttachment = vk::AttachmentDescription(
        vk::AttachmentDescriptionFlags(),
        depthFormat,
        msaaSamples,
        vk::AttachmentLoadOp::eClear,
        vk::AttachmentStoreOp::eDontCare,
        vk::AttachmentLoadOp::eDontCare,
        vk::AttachmentStoreOp::eDontCare,
        vk::ImageLayout::eUndefined,
        vk::ImageLayout::eDepthStencilAttachmentOptimal
    );

    auto colorAttachmentResolve = vk::AttachmentDescription(
        vk::AttachmentDescriptionFlags(),
        imageFormat,
        vk::SampleCountFlagBits::e1,
        vk::AttachmentLoadOp::eDontCare,
        vk::AttachmentStoreOp::eStore,
        vk::AttachmentLoadOp::eDontCare,
        vk::AttachmentStoreOp::eDontCare,
        vk::ImageLayout::eUndefined,
        vk::ImageLayout::eColorAttachmentOptimal
    );

    auto colorAttachmentRef = vk::AttachmentReference(0, vk::ImageLayout::eColorAttachmentOptimal);
    auto depthAttachmentRef = vk::AttachmentReference(1, vk::ImageLayout::eDepthStencilAttachmentOptimal);
    auto colorAttachmentResolveRef = vk::AttachmentReference(2, vk::ImageLayout::eColorAttachmentOptimal);

    auto subpass = vk::SubpassDescription(
        vk::SubpassDescriptionFlags(),
        vk::PipelineBindPoint::eGraphics,
        0, nullptr,                             // input attachment
        1, &colorAttachmentRef,                 // color attachment
        &colorAttachmentResolveRef,             // resolve attachment
        &depthAttachmentRef                     // depth attachment
    );

    auto dependency = vk::SubpassDependency(
        /*srcSubpass*/      static_cast<uint32_t>(0),
        /*dstSubpass*/      static_cast<uint32_t>(VK_SUBPASS_EXTERNAL),
        /*srcStageMask*/    vk::PipelineStageFlagBits::eColorAttachmentOutput | vk::PipelineStageFlagBits::eEarlyFragmentTests,
        /*dstStageMask*/    vk::PipelineStageFlagBits::eColorAttachmentOutput | vk::PipelineStageFlagBits::eEarlyFragmentTests,
        /*srcAccessMask*/   vk::AccessFlagBits(0),
        /*dstAccessMask*/   vk::AccessFlagBits::eColorAttachmentWrite | vk::AccessFlagBits::eDepthStencilAttachmentWrite
    );

    std::array<vk::AttachmentDescription, 3> attachments = { colorAttachment, depthAttachment, colorAttachmentResolve };
    auto renderPassInfo = vk::RenderPassCreateInfo(
        vk::RenderPassCreateFlags(),
        static_cast<uint32_t>(attachments.size()), attachments.data(),
        1, &subpass,
        1, &dependency
    );

    try {
        renderPass = device.createRenderPass(renderPassInfo);
    }
    catch (vk::SystemError err) {
        throw std::runtime_error("failed to create render pass!");
    }
}

The initialization of the second Render Pass belonging to DearImGui looks like this:

void imGuiInstance::initRenderPass(vk::Format _format) {
    auto attachment = vk::AttachmentDescription(
        vk::AttachmentDescriptionFlags(),
        _format,
        vk::SampleCountFlagBits::e1,
        vk::AttachmentLoadOp::eLoad,
        vk::AttachmentStoreOp::eStore,
        vk::AttachmentLoadOp::eDontCare,
        vk::AttachmentStoreOp::eDontCare,
        vk::ImageLayout::eColorAttachmentOptimal,
        vk::ImageLayout::ePresentSrcKHR
    );

    auto colorAttachment = vk::AttachmentReference(0, vk::ImageLayout::eColorAttachmentOptimal);
    
    auto subpass = vk::SubpassDescription(
        vk::SubpassDescriptionFlags(),
        vk::PipelineBindPoint::eGraphics,
        0, nullptr,
        1, &colorAttachment
    );

    auto dependency = vk::SubpassDependency(
        /*srcSubpass*/      static_cast<uint32_t>(VK_SUBPASS_EXTERNAL),
        /*dstSubpass*/      static_cast<uint32_t>(0),
        /*srcStageMask*/    vk::PipelineStageFlagBits::eColorAttachmentOutput,
        /*dstStageMask*/    vk::PipelineStageFlagBits::eColorAttachmentOutput,
        // This might need to be changed to vk::AccessFlagBits::eColorAttachmentWrite
        /*srcAccessMask*/   vk::AccessFlagBits(0),
        /*dstAccessMask*/   vk::AccessFlagBits::eColorAttachmentWrite
    );

    auto renderPassCreateInfo = vk::RenderPassCreateInfo(
        vk::RenderPassCreateFlags(),
        1, &attachment,
        1, &subpass,
        1, &dependency
    );

    try {
        renderPass = device.createRenderPass(renderPassCreateInfo);
    }
    catch (vk::SystemError err) {
        throw std::runtime_error("failed to create ImGui renderpass");
    }
}

PS: I know it would be easier to solve this issue with one Render Pass and subpasses, but I want to keep DearImGui seperate from the rest of the program.

Edit: Due to @Jhericos answer I have changed the dependencies in both Render Passes, now the first Render Pass has:

auto firstDependency = vk::SubpassDependency(
    static_cast<uint32_t>(VK_SUBPASS_EXTERNAL),
    static_cast<uint32\_t>(0), 
    vk::PipelineStageFlagBits::eColorAttachmentOutput | vk::PipelineStageFlagBits::eEarlyFragmentTests,
     vk::PipelineStageFlagBits::eColorAttachmentOutput | vk::PipelineStageFlagBits::eEarlyFragmentTests, 
    vk::AccessFlagBits(0), 
    vk::AccessFlagBits::eColorAttachmentWrite | vk::AccessFlagBits::eDepthStencilAttachmentWrite 
);

auto secondDependency = vk::SubpassDependency(
    /*srcSubpass*/      static_cast<uint32_t>(0),
    /*dstSubpass*/      static_cast<uint32_t>(VK_SUBPASS_EXTERNAL),
    /*srcStageMask*/    vk::PipelineStageFlagBits::eColorAttachmentOutput | vk::PipelineStageFlagBits::eEarlyFragmentTests,
    /*dstStageMask*/    vk::PipelineStageFlagBits::eColorAttachmentOutput | vk::PipelineStageFlagBits::eEarlyFragmentTests,
    /*srcAccessMask*/   vk::AccessFlagBits::eColorAttachmentWrite | vk::AccessFlagBits::eDepthStencilAttachmentWrite,
    /*dstAccessMask*/   vk::AccessFlagBits::eColorAttachmentRead
);

And the second Render Pass has:

auto firstDependency = vk::SubpassDependency(
    /*srcSubpass*/      static_cast<uint32_t>(VK_SUBPASS_EXTERNAL),
    /*dstSubpass*/      static_cast<uint32_t>(0),
    /*srcStageMask*/    vk::PipelineStageFlagBits::eColorAttachmentOutput,
    /*dstStageMask*/    vk::PipelineStageFlagBits::eColorAttachmentOutput,
    // This might need to be changed to vk::AccessFlagBits::eColorAttachmentWrite
    /*srcAccessMask*/   vk::AccessFlagBits(0),
    /*dstAccessMask*/   vk::AccessFlagBits::eColorAttachmentRead
);

auto secondDependecy = vk::SubpassDependency(
    static_cast<uint32_t>(0),
    static_cast<uint32_t>(VK_SUBPASS_EXTERNAL),
    vk::PipelineStageFlagBits::eColorAttachmentOutput,
    vk::PipelineStageFlagBits::eColorAttachmentOutput,
    vk::AccessFlagBits::eColorAttachmentRead,
    vk::AccessFlagBits::eColorAttachmentWrite
);

The errors still persist and now I get even newer errors, although I think this is due to ImGuis own initialization:

validation layer [XXXXX]: Validation Error: [ VUID-vkCmdDrawIndexed-renderPass-02684 ] Object 0: handle = 0x564fdd0000000069, type = VK_OBJECT_TYPE_RENDER_PASS; Object 1: handle = 0xea7170000000031, type = VK_OBJECT_TYPE_RENDER_PASS; | MessageID = 0x8cb637c2 | vkCmdDrawIndexed: RenderPasses incompatible between active render pass w/ VkRenderPass 0x564fdd0000000069[] with a dependencyCount of 1 and pipeline state object w/ VkRenderPass 0xea7170000000031[] with a dependencyCount of 2. The Vulkan spec states: The current render pass must be compatible with the renderPass member of the VkGraphicsPipelineCreateInfo structure specified when creating the VkPipeline bound to VK_PIPELINE_BIND_POINT_GRAPHICS (https://vulkan.lunarg.com/doc/view/1.3.239.0/windows/1.3-extensions/vkspec.html#VUID-vkCmdDrawIndexed-renderPass-02684)

But I still don't quite understand why two dependencies should be necessary. Why isn't it enough for the first Render Pass to create a dependecy that waits for every color writing operation to be done before letting the next Render Pass read from its outputs?

In my mind this is achieved by setting the dstSubpass as VK_SUBPASS_EXTERNAL, the srcAccessMask to vk::AccessFlagBits::eColorAttachmentWrite and the dstAccessMask to vk::AccessFlagBits::eColorAttachmentRead. Similarly in the second Render Pass we would need to define a dependency by setting srcSubpass as VK_SUBPASS_EXTERNAL, the srcAccessMask to vk::AccessFlagBits::eColorAttachmentWrite and the dstAccessMask to vk::AccessFlagBits::eColorAttachmentRead.

What am I missing?

Answer 1

I'm trying to integrate DearImGui with my Vulkan application in a way that the application and DearImGui use seperate Render Passes, but the same Command Buffer.

That seems questionable, because in many setups the scene rendering command buffers and the UI command buffers are going to have vastly different rates of change, so having to re-record a single giant command buffer when either the UI or the rendered scene changes is wasteful.

Aside from that, each of your renderpasses should have at least two subpass dependencies, VK_SUBPASS_EXTERNAL -> 0, and 0 -> VK_SUBPASS_EXTERNAL. I suspect the lack of those is part of your problem.

PS: I know it would be easier to solve this issue with one Render Pass and subpasses, but I want to keep DearImGui seperate from the rest of the program.

No, actually, you want the UI stuff in a subpass even less than you want it in the same command buffer as rendering.

However, what would be easier overall would be to switch to dynamic rendering and junk all your renderpass, framebuffer and subpass code entirely. You have to manage image transitions manually, but it's way less code than renderpasses impose.