CODEWITHSUNDEEP
phpjsonutf-8mb-convert-encoding
fractal5
fractal5

Reputation: 2132

Having en-dash at the end of the string doesn't allow json_encode

I am trying to extract n characters from a string using substr($originalText,0,250);

The nth character is an en-dash. So I get the last character as †when I view it in notepad. In my editor, Brackets, I can't even open the log file it since it only supports UTF-8 encoding.

I also cannot run json_encode on this string.

However, when I use substr($originalText,0,251), it works just fine. I can open the log file and it shows an en-dash instead of â€. json_encode also works fine.

I can use mb_convert_encoding($mystring, "UTF-8", "Windows-1252") to circumvent the problem, but could anyone tell me why having these characters at the end specifically causes an error? Moreover, on doing this, my log file shows †in brackets, which is confusing too.

My question is why is having the en-dash at the end of the string, different from having it anywhere else (followed by other characters).

Hopefully my question is clear, if not I can try to explain further.

Thanks.

Upvotes: 0

Views: 928

Answers (2)

pid
pid

Reputation: 11607

UTF-8 uses so-called surrogates which extend the codepage beyond ASCII to accomodate many more characters.

A single UTF-8 character may be coded into one, two, three or four bytes, depending on the character.

You cut the string right in the middle of a multi-byte character:

[<-character->]
[byte-0|byte-1]
       ^
      You cut the string right here in the middle!


[<-----character---->]
[byte-0|byte-1|byte-2]
       ^      ^
      Or anywhere here if it's 3 bytes long.

So the decoder has the first byte(s) but can't read the entire character because the string ends prematurely.

This causes all the effects you are witnessing.

The solution to this problem is here in Dezza's answer.

Upvotes: 4

Dezza
Dezza

Reputation: 1094

Pid's answer gives an explanation for why this is happening, this answer just looks at what you can do about it...

Use mb_substr()

The multibyte string module was designed for exactly this situation, and provides a number of string functions that handle multibyte characters correctly. I suggest having a look through there as there are likely other ones that you will need in other places of your application.

You may need to install or enable this module if you get a function not found error. Instructions for this are platform dependent and out-of-scope for this question.

The function you want for the case in your question is called mb_substr() and is called the same as you would use substr(), but has other optional arguments.

Upvotes: 5

Related Questions