CODEWITHSUNDEEP
winapiutf-8codepageswindows-11
Rene
Rene

Reputation: 3776

WIndows 11 Default API and UTF Encoding

In Windows 10 and earlier, I have been able to transfer strings in my local codepage 1250 or with CP_ACP with the following code successfully to UTF-8. But in Windows 11, this does no longer work with CP_ACP (while 1250 still works). It seems that the default codepage is now 65001, which cannot be translated to UTF-8 this way. The result is simply false.

The reason is probably, that my string "Öf" in the example is not properly encoded in 65001. Now I have a big project, where the user enters strings and various third-party play a role, which all seem to deliver strings in 1250, or the current codepage of a non-European user.

Why is that? And what to do?

#include <Windows.h>

#include <cstdio>

int main()
{
    printf("UTF Conversation Test\n");

    char line[1000];
    WCHAR uline[1000];
    char uline1[1000];

    line[0] = 214;
    line[1] = 104;
    line[2] = 0;

    char *s1 = line;
    while (*s1 != 0)
    {
        printf("%10x %d\n", (int)*s1, (int)*s1);
        s1++;
    }
    printf("\n");

    MultiByteToWideChar(1250, 0, line, -1, uline, 1000);
    // MultiByteToWideChar(CP_ACP, 0, line, -1, uline, 1000);

    WCHAR* s2 = uline;

    while (*s2 != 0)
    {
        printf("%10x %d\n", (int)*s2, (int)*s2);
        s2++;
    }
    printf("\n");

    WideCharToMultiByte(CP_UTF8, 0, uline, -1, uline1, 1000, 0, 0);

    char *s3 = uline1;

    while (*s3 != 0)
    {
        printf("%10x %d\n", (int)*s3, (int)*s3);
        s3++;
    }
}

Upvotes: 2

Views: 9365

Answers (1)

Rene
Rene

Reputation: 3776

It turns out that Windows 11 activates Beta support for UTF-8 system-wide by default. This means that any programs that do not store strings in Unicode internally will have to translate to UTF-8 and back for using Windows services like screen output of characters. Even worse, some of their dialogs may stop to show local characters correctly. One solution is to disable this Beta support in the Administrative settings for the time and region.

Upvotes: 4

Related Questions