CODEWITHSUNDEEP
c#.netutf-8character-encodingutf-16
Mike Marynowski
Mike Marynowski

Reputation: 3429

Maximum UTF-8 string size given UTF-16 size

What is the formula for determining the maximum number of UTF-8 bytes required to encode a given number of UTF-16 code units (i.e. the value of String.Length in C# / .NET)?

I see 3 possibilities:

  1. # of UTF-16 code units x 2

  2. # of UTF-16 code units x 3

  3. # of UTF-16 code units x 4

A UTF-16 code point is represented by either 1 or 2 code units, so we just need to consider the worst case scenario of a string filled with one or the other. If a UTF-16 string is composed entirely of 2 code unit code points, then we know the UTF-8 representation will be at most the same size, since the code points take up a maximum of 4 bytes in both representations, thus worst case is option (1) above.

So the interesting case to consider, which I don't know the answer to, is the maximum number of bytes that a single code unit UTF-16 code point can require in UTF-8 representation.

If all single code unit UTF-16 code points can be represented with 3 UTF-8 bytes, which my gut tells me makes the most sense, then option (2) will be the worst case scenario. If there are any that require 4 bytes then option (3) will be the answer.

Does anyone have insight into which is correct? I'm really hoping for (1) or (2) as (3) is going to make things a lot harder :/

UPDATE

From what I can gather, UTF-16 encodes all characters in the BMP in a single code unit, and all other planes are encoded in 2 code units.

It seems that UTF-8 can encode the entire BMP within 3 bytes and uses 4 bytes for encoding the other planes.

Thus it seems to me that option (2) above is the correct answer, and this should work:

string str = "Some string";
int maxUtf8EncodedSize = str.Length * 3;

Does that seem like it checks out?

Upvotes: 5

Views: 3502

Answers (2)

Timmmm
Timmmm

Reputation: 96556

The worst case for a single UTF-16 word is U+FFFF which in UTF-16 is encoded just as-is (0xFFFF) Cyberchef. In UTF-8 it is encoded to ef bf bf (three bytes).

The worst case for two UTF-16 words (a "surrogate pair") is U+10FFFF which in UTF-16 is encoded as 0xDBFF DFFF. In UTF-8 it is encoded to f3 cf bf bf (four bytes).

Therefore the worst case is a load of U+FFFF's which will convert a UTF-16 string of length 2N bytes to a UTF-8 string of length 3N bytes.

So yes, you are correct. I don't think you need to consider stuff like glyphs because that sort of thing is done after decoding from UTF8/16 to code points.

Upvotes: 4

JasonTrue
JasonTrue

Reputation: 19609

Properly formed UTF-8 can be up to 4 bytes per Unicode codepoint.

UTF-16-encoded characters can be up to 2 16-bit sequences per Unicode codepoint.

Characters outside the basic multilingual plane (including emoji and languages that were added to more recent versions of Unicode) are represented in up to 21 bits, which in the UTF-8 format results in 4 byte sequences, which turn out to also take up 4 bytes in UTF-16.

However, there are some environments that do things weirdly. Since UTF-16 characters outside the basic multilingual plane take up to 2 16-bit sequences (they're detectible because they're always 16 bit sequences in the range U+D800 to U+DFFF), some mistaken UTF-8 implementations, usually referred to as CESU-8, that convert those UTF-8 sequences into two 3-byte UTF-8 sequences, for a total of six bytes per UTF-32 codepoint. (I believe some early Oracle DB implementations did this, and I'm sure they weren't the only ones).

There's one more minor wrench in things, which is that some glyphs are classified as combining characters, and multiple UTF-16 (or UTF-32) sequences are used when determining what gets displayed on the screen, but I don't think that applies in your case.

Based on your edit, it looks like you're trying to estimate the maximum length of .Net encoding conversion. String Length measures the total number of Chars, which are a count of UTF-16 codepoints. As a worst-case estimate, therefore, I believe you can safely estimate count(Char) * 3, because the non-BMP characters will be count(Char) * 2 yielding 4 bytes as UTF-8.

If you want to get the total number of UTF-32 codepoints represented, you should be able to do something like

var maximumUtf8Bytes = System.Globalization.StringInfo(myString).LengthInTextElements * 4;

(My C# is a bit rusty as I haven't used a .Net environment much in the last few years, but I think that does the trick).

Upvotes: 2

Related Questions