CODEWITHSUNDEEP
c#.netencoding
Alex Zhukovskiy
Alex Zhukovskiy

Reputation: 10015

Get a substring that is at most N bytes UTF8 string

I'm using some API that requires that input string is a valid UTF8 string with maximum length of 4096 bytes.

I had following function to trim the extra characters:

private static string GetTelegramMessage(string message)
{
    const int telegramMessageMaxLength = 4096; // https://core.telegram.org/method/messages.sendMessage#return-errors
    const string tooLongMessageSuffix = "...";

    if (message == null || message.Length <= 4096)
    {
        return message;
    }

    return message.Remove(telegramMessageMaxLength - tooLongMessageSuffix.Length) + tooLongMessageSuffix;
}

It didn't work well because characters != bytes and UTF16 chars != UTF8 chars.

So basically I need to convert my C# UTF16 string into UTF8 string with fixed length. I do

var bytes = Encoding.UTF8.GetBytes(myString);
// now I need to get first N characters with overall bytes size less than 4096 bytes

I can express my need in Rust (working example below):

fn main() {
    let foo = format!("{}{}", "ᚠᛇᚻ᛫ᛒᛦᚦ᛫ᚠᚱᚩᚠᚢᚱ᛫ᚠᛁᚱᚪ᛫ᚷᛖᚻᚹᛦᛚᚳᚢᛗ Uppen Sevarne staþe, sel þar him þuhte", (1..5000).map(|_| '1').collect::<String>());
    println!("{}", foo.len());
    let message = get_telegram_message(&foo);
    println!("{}", message);
    println!("{}", message.chars().count()); // 4035
    println!("{}", message.len()); // 4096
}

pub fn get_telegram_message(foo: &str) -> String {
    const PERIOD: &'static str = "...";
    const MAX_LENGTH: usize = 4096;
    let message_length = MAX_LENGTH - PERIOD.len();

    foo.chars()
        .map(|c| (c, c.len_utf8())) // getting length for evey char
        .scan((0, '\0'), |(s, _), (c, size)| {
            *s += size; //  running total for all previosely seen characters
            Some((*s, c))
        })
        .take_while(|(len, _)| len <= &message_length) // taking while running total is less than maximum message size
        .map(|(_, c)| c)
        .chain(PERIOD.chars()) // add trailing ellipsis
        .collect() // building a string
}

https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=471ad0cbe9b0b01b50ec250d17dea233

The problem here is that I don't have chars() iterator in C# that allows me to treat bytes sequence as UTF8 characters.

I've played with Encoding.UTF8 a bit but I didn't find appropriate APIs to perform this task.

Linked articles is somehow related to my question, but first answer it just very bad, the second one reimplement UTF8 iterator (that's what I called IEnumerable<long> below). Since I know how to implement it, my question about builtin function to perform this task so neither of linked answers answers that.

Upvotes: 2

Views: 781

Answers (1)

canton7
canton7

Reputation: 42225

I think Encoder.Convert is probably the method you're after.

I interpreted the question as meaning

I have a string, which will be turned into UTF-8 bytes. I want to trim it such that its UTF-8 encoding is a maximum of 4096 bytes, but I want to make sure I don't trim it in the middle of a UTF-8 codepoint.

private static string GetTelegramMessage(string message)
{
    const int telegramMessageMaxLength = 4096; // https://core.telegram.org/method/messages.sendMessage#return-errors
    const string tooLongMessageSuffix = "...";

    if (string.IsNullOrEmpty(message) || Encoding.UTF8.GetByteCount(message) <= telegramMessageMaxLength)
    {
        return message;
    }

    var encoder = Encoding.UTF8.GetEncoder();
    byte[] buffer = new byte[telegramMessageMaxLength - Encoding.UTF8.GetByteCount(tooLongMessageSuffix)];
    char[] messageChars = message.ToCharArray();
    encoder.Convert(
        chars: messageChars,
        charIndex: 0,
        charCount: messageChars.Length,
        bytes: buffer,
        byteIndex: 0,
        byteCount: buffer.Length,
        flush: false,
        charsUsed: out int charsUsed,
        bytesUsed: out int bytesUsed,
        completed: out bool completed);

    // I don't think we can return message.Substring(0, charsUsed)
    // as that's the number of UTF-16 chars, not the number of codepoints
    // (think about surrogate pairs). Therefore I think we need to
    // actually convert bytes back into a new string
    return Encoding.UTF8.GetString(bytes, 0, bytesUsed) + tooLongMessageSuffix;
}

Upvotes: 1

Related Questions