CODEWITHSUNDEEP
unicodeicutransliteration
Jacob Zwiers
Jacob Zwiers

Reputation: 1102

Transliteration on Unicode LATIN LETTERS "WITH STROKE"

Feeding the rule "NFD; [:Nonspacing Mark:] Remove; NFC" into the ICU Transliterator demo, the character Ø (\u00d8 == LATIN CAPITAL LETTER O WITH STROKE) remains as-is (i.e. the STROKE is not stripped).

In the list of non-marking spaces (Category Mn), I cannot find anything named COMBINING DIAGONAL STROKE akin to the COMBINING SHORT STROKE OVERLAY (\u0335) or COMBINING LONG STROKE OVERLAY (\u0336).

However, I do find COMBINING SHORT SOLIDUS OVERLAY (\u0337) and COMBINING LONG SOLIDUS OVERLAY (\u0338). They appear similar, but render as much thicker lines in my browser when combined with o and O.

The Unicode data I accessed for \u00d8 does not provide a decomposition for that character.

At the same time, the ICU Collator Demo will collate each of ø, o, Ø, O, o\u0337 and O\u0338 to the same code point using a Primary (Level = 1 = Base Letter) Collator.

Does this mean that the locale of Collator used in the Demo has been set up to identify the base character in a way where the Unicode spec is silent?

If so, do I need to a custom Rule Based Transliterator if I want to strip the STROKE from LATIN [CAPITAL, SMALL] LETTER * characters on transliteration?

Upvotes: 1

Views: 3018

Answers (3)

user3389160
user3389160

Reputation: 1

This transform along with replaceAll works even for removing the Ø and other characters.

String id = "Accents-Any;NFD;[:Nonspacing Mark:] Remove; NFC";
System.out.println(latin.replaceAll("[^\\w]",""));

Upvotes: 0

Steven R. Loomis
Steven R. Loomis

Reputation: 4350

See the following. The Latin-ASCII transliterator went into ICU 4.6. As you noted, the collation demo uses UCA / CLDR tailorings which have O versus slashed-O as base letter differences, this is not the same question as whether there's a decomposition. "w" doesn't decompose into "v + v" either. The decompositions have to do with whether there were existing encodings which represent characters in two different ways.

Upvotes: 2

dan04
dan04

Reputation: 90995

Yes. For some reason, the letter Ø does not have a decomposition, so you have to handle it manually.

Upvotes: 1

Related Questions