Convert any input DDS into another DDS format in DirectXTex

Question

I want to create a function to convert the input dds file to another format (also DDS) function signature will be like this

std::vector<byte> ConvertDDS(const byte* src, const size_t src_len, const DXGI_FORMAT format)

in here src will be a pointer to the dds file content (full dds), and format will be the format that we want to convert the src into, the function return a vector of bytes that hold the converted dds.
as I said before the input DDS can be any format (I mean not any, but we better guess that it will be any) it can be compressed or uncompressed so we need to handle these two cases as well What would be the right and correct way to write this function?
Currently I have this code, it will convert the dds, but the result aren't what I except them to be:

• first of all most of the input dds have transparent background but this method can't keep it and will replace the background with just white
• second the converted texture will only have one alpha channel that bically make the whole texture useless

Here is a working example of my current code

#include <iostream>
#include <fstream>
#include <vector>
#include <DirectXTex.h>

void throw_error(std::string err, HRESULT hr)
{
    LPWSTR errorText = nullptr;
    DWORD messageLength = FormatMessageW(FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
        nullptr, hr & 0x0000FFFF, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), (LPWSTR)&errorText, 0, nullptr);
    if (messageLength > 0)
    {
        std::wstring wErrorText(errorText);
        std::string errorString(wErrorText.begin(), wErrorText.end());
        LocalFree(errorText);
        throw std::invalid_argument(err + ": " + errorString);
    }
}

std::vector<byte> ConvertDDS(const byte* src, const size_t src_len, const DXGI_FORMAT format)
{
    DirectX::TEX_FILTER_FLAGS dwFilter = DirectX::TEX_FILTER_DEFAULT;
    DirectX::TEX_FILTER_FLAGS dwSRGB = DirectX::TEX_FILTER_DEFAULT;
    DirectX::TEX_FILTER_FLAGS dwConvert = DirectX::TEX_FILTER_DEFAULT;
    DirectX::TEX_FILTER_FLAGS dwFilterOpts = DirectX::TEX_FILTER_DEFAULT;
    DirectX::TexMetadata meta;

    // Initialize COM (needed for WIC)
    HRESULT hr = CoInitializeEx(nullptr, COINIT_MULTITHREADED);
    if (FAILED(hr))
        throw_error("Failed to initialize COM", hr);

    std::unique_ptr<DirectX::ScratchImage> image(new (std::nothrow) DirectX::ScratchImage);
    // load texture from memory, supported types are DDS, PNG and TGA
    hr = DirectX::LoadFromDDSMemory(src, src_len, DirectX::DDS_FLAGS_ALLOW_LARGE_FILES, &meta, *image);

    if (FAILED(hr))
        throw_error("Can't load texture from memory", hr);

    // If format doesn't match convert it
    if (meta.format != format)
    {
        // --- Planar ------------------------------------------------------------------
        if (DirectX::IsPlanar(meta.format))
        {
            auto img = image->GetImage(0, 0, 0);
            assert(img);
            const size_t nimg = image->GetImageCount();

            std::unique_ptr<DirectX::ScratchImage> timage(new (std::nothrow) DirectX::ScratchImage);

            hr = ConvertToSinglePlane(img, nimg, meta, *timage);
            if (FAILED(hr))
                throw_error(" FAILED [converttosingleplane]", hr);

            auto& tinfo = timage->GetMetadata();

            meta.format = tinfo.format;

            assert(meta.width == tinfo.width);
            assert(meta.height == tinfo.height);
            assert(meta.depth == tinfo.depth);
            assert(meta.arraySize == tinfo.arraySize);
            assert(meta.mipLevels == tinfo.mipLevels);
            assert(meta.miscFlags == tinfo.miscFlags);
            assert(meta.dimension == tinfo.dimension);

            image.swap(timage);
        }

        // --- Decompress --------------------------------------------------------------
        if (DirectX::IsCompressed(meta.format))
        {
            std::unique_ptr<DirectX::ScratchImage> nimage(new (std::nothrow) DirectX::ScratchImage);
            hr = DirectX::Decompress(
                image->GetImages(),
                image->GetImageCount(),
                meta,
                DXGI_FORMAT_UNKNOWN,
                *nimage
            );
            if (FAILED(hr))
                throw_error("Can't decompress dds texture", hr); // decompress new texture failed
            meta.format = nimage->GetMetadata().format;
            image.swap(nimage);
        }

        if (meta.format != format)
        {
            // If the original file is compressed we compress the src
            // Else we just convert it
            if (DirectX::IsCompressed(format))
            {
                std::unique_ptr<DirectX::ScratchImage> nimage(new (std::nothrow) DirectX::ScratchImage);
                hr = DirectX::Compress(
                    image->GetImages(),
                    image->GetImageCount(),
                    meta,
                    format,
                    DirectX::TEX_COMPRESS_FLAGS::TEX_COMPRESS_DEFAULT,
                    DirectX::TEX_THRESHOLD_DEFAULT,
                    *nimage
                );
                if (FAILED(hr))
                    throw_error("Can't compress dds texture", hr); // convert new texture failed
                image.swap(nimage);
            }
            else
            {
                std::unique_ptr<DirectX::ScratchImage> nimage(new (std::nothrow) DirectX::ScratchImage);
                hr = DirectX::Convert(
                    image->GetImages(),
                    image->GetImageCount(),
                    meta,
                    format,
                    dwFilter | dwFilterOpts | dwSRGB | dwConvert,
                    DirectX::TEX_THRESHOLD_DEFAULT,
                    *nimage
                );
                if (FAILED(hr))
                    throw_error("Can't convert dds texture", hr); // convert new texture failed
                image.swap(nimage);
            }
            meta.format = image->GetMetadata().format;
        }
    }

    std::unique_ptr<DirectX::Blob> blob(new (std::nothrow) DirectX::Blob);
    DirectX::SaveToDDSMemory(
        image->GetImages(),
        image->GetImageCount(),
        image->GetMetadata(),
        DirectX::DDS_FLAGS_NONE,
        *blob.get()
    );
    std::vector<byte> result(blob->GetBufferSize());
    std::memcpy(result.data(), blob->GetBufferPointer(), blob->GetBufferSize());
    return result;
}

this code does seem to be working, but output as I said dont have the transparent background and overall looks wrong... what should I do? I'm not very familer with DirectXTex API so I will be happy if someone can help me.

I tried to convert a DDS texture with format DXGI_FORMAT_B8G8R8A8_UNORM to DXGI_FORMAT_R8_UNORM but failed.
input had transparency but output only had a white background
the content of the texture also were wrong after conversation.

Answer 1

What behavior do you want for going from DXGI_FORMAT_B8G8R8A8_UNORM to DXGI_FORMAT_R8_UNORM?

The default behavior for an RGB to R conversion is to use luminance to convert the RGB to a greyscale and store that in the RED channel. There's no place to put a transparency channel in the target.

Are you expecting the alpha channel to 'mask' the image here? If so, what's the 'background color'?

If you want to keep the alpha channel you'd need to use DXGI_FORMAT_R8A8_UNORM.

DirectXTex does support other ways of mapping the channels. You can do channel swizzles, or you can use flags like TEX_FILTER_RGB_COPY_RED, TEX_FILTER_RGB_COPY_GREEN, or TEX_FILTER_RGB_COPY_BLUE. See Filter flags.

For examples of doing arbitrary channel swizzles, see texconv.