CODEWITHSUNDEEP
unicodeencodingfontscharacterutf
chenxinlong
chenxinlong

Reputation: 1837

Where are the unicode characters on the disk and what's the mapping process?

There are several unicode relevant questions has been confusing me for some time.

For these reasons as follow I think the unicode characters are existed on disk.

  1. Execute echo "\u6211" in terminal, it will print the glyph corresponding to the unicode code point U+6211.
  2. There's a concept of UCD (unicode character database), and We can download it's latest version. UCD latest
  3. Some new version unicode characters like latest emojis can not display on my mac until I upgrade macOS version.

So if the unicode characters does existed on the disk , then ：

  1. Where is it ?
  2. How can I upgrade it ?
  3. What's the process of mapping the unicode code point to a glyph ?
  4. If I use a specific font, then what's the process of mapping the unicode code point to a glyph ?

If not, then what's the process of mapping the unicode code point to a glyph ?

It will very appreciated if someone could shed light on these problems.

Upvotes: 1

Views: 431

Answers (2)

daxim
daxim

Reputation: 39158

Execute echo "\u6211" in terminal, it will print the glyph corresponding to the unicode code point U+6211.

That's echo -e in bash.

› echo "\u6211"
\u6211
› echo -e "\u6211"
我

Where is it ?

In the font file.

Some new version unicode characters like latest emojis can not display on my mac until I upgrade macOS version. How can I upgrade it ?

Installing/upgrading a suitable font with the emojis should be enough. I don't have macOS, so I cannot verify this.

I use "Noto Color Emoji" version 2.011/20180424, it works fine.

What's the process of mapping the unicode code point to a glyph ?

The application (e.g. text editor) provides the font rendering subsystem (Quartz? on macOS) with Unicode text and a font name. The font renderer analyses the codepoints of the text and decides whether this is simple text (e.g. Latin, Chinese, stand-alone emojis) or complex text (e.g. Latin with many marks, Thai, Arabic, emojis with zero-width joiners). The renderer finds the corresponding outlines in the font file. If the file does not have the required glyph, the renderer may use a similar font, or use a configured fallback font for a poor substitute (white box, black question mark etc.). Then the outlines undergo shaping to compose a complex glyph and line-breaking. Finally, the font renderer hands off the result to the display system.

Apart from the shaping, very little of this has to do with Unicode or encoding. Font rendering already used to work that way before Unicode existed, of course font files and rendering was much simpler 30 years ago. Encoding only matters when someone wants to load or save text from an application.

Summary: investigate

  • Truetype/Opentype font editing software so you can see what's contained in the files
  • font renderers, on Linux look at the libraries pango and freetype.

Upvotes: 1

Tom Blodget
Tom Blodget

Reputation: 20812

Generally speaking, operating system components that use text use the Unicode character set. In particular, font files use the Unicode character set. But, not all font files support all the Unicode codepoints.

When a codepoint is not supported by one font, the system might fallback to another that does. This is particularly true of web browsers. But ultimately if the codepoint is not supported, an unfilled rectangle is rendered. (There is no character for that because it's not a character. In fact, if you were able to copy and paste it as text, it should be the original character that couldn't be rendered.)

In web development, the web page can either supply or give the location of fonts that should work for the codepoints it uses.

Other programs typically use the operating system's rendering facilities and therefore the fonts available through it. How to install a font in an operating system is not a programming question (unless you are including a font in an installer for your program). For more information on that, you could see if the question fits with the Ask Different (Apple) Stack Exchange site.

Upvotes: 0

Related Questions