CODEWITHSUNDEEP
objective-cunicodeutf-8
brian
brian

Reputation: 157

Unihan: combining UTF-8 chars

I am using data that involves Chinese Unihan characters in an Objective-C app. I am using a voice recognition program (cmusphinx) that returns a phrase from my data. It returns UTF-8 characters and when returning a Chinese character (which is three bytes) it separates it into three separate characters. Example: When I want 人 to, I see: ‰∫∫. This is the proper in coding (E4 BA BA), but my code sees the returned value as three seperate characters rather than one.

Actually, my function is receiving the phrase as an NSString, (due to a wrap around) which uses UTF-16. I tried using Objective-C's built in conversion methods (to UTF-8 and from UTF-16), but these keep my string as three characters.

How can I decode these three separate characters into the one utf-8 codepoint for the Chinese character?

Or how can I properly encode it? This is code fragment dealing with the cstring returned from sphinx and its encoding to a NSString:

const char * hypothesis = ps_get_hyp(pocketSphinxDecoder, &recognitionScore, &utteranceID);
NSString *hypothesisString = [[NSString alloc] initWithCString:hypothesis encoding:NSMacOSRomanEncoding];

Upvotes: 0

Views: 736

Answers (1)

Lily Ballard
Lily Ballard

Reputation: 185751

Edit: From looking at the addition to your post, you actually do have control over the string encoding. In that case, why are you creating the string with NSMacOSRomanEncoding when you're expecting utf-8? Just change that to NSUTF8StringEncoding.

It sounds like what you're saying is you're being given an NSString that contains UTF-8 data that's being interpreted as a single-byte encoding (e.g. ISO-Latin-1, MacRoman, etc). I'm assuming here that you have no control over the code that creates the NSString, because if you did then the solution is just to change the encoding it's initializing with.

In any case, what you're asking for is a way to take the data in the string and convert it back to UTF-8. You can do this by creating an NSData from the NSString using whatever encoding its was originally created with (you need to know this much, at least, or it won't work), and then you can create a new NSString from the same data using UTF-8.

From the example character you gave (人) it looks like it's being interpreted as MacRoman, so lets go with that. The following code should convert it back:

- (NSString *)fixEncodingOfString:(NSString *)input {
    CFStringEncoding cfEncoding = kCFStringEncodingMacRoman;
    NSStringEncoding encoding = CFStringCovnertEncodingToNSStringEncoding(cfEncoding);
    NSData *data = [input dataUsingEncoding:encoding];
    if (!data) {
        // the string wasn't actually in MacRoman
        return nil;
    }
    NSString *output = [[[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding] autorelease];
}

Upvotes: 1

Related Questions