19
$\begingroup$

Background:

Recently I'm trying to deal with some Chinese characters which belongs to the CJK Unified Ideographs Extension B block. The motive comes from some further interest following a former question, and some ideas of using Mathematica for typesetting ancient Chinese scriptures.

Edit:

Here is some sample code which looks fine at first sight ( The Unicodes of the three characters involved here are x20027, x200D5 and x20137. The setting of the $SystemCharacterEncoding of my Mathematica is "CP936". ):

enter image description here

But after saving then closing the notebook, I re-opened it, without touching anything, the Front End displays weird now (and note the attendance of the syntax errors):

enter image description here

It seems the Front End is not able to correctly render them.

Furthermore, although I can create a textcell looking fine:

enter image description here

whose CellExpression is:

Cell["\:d840\:dc27\:d840\:dcd5\:d840\:dd37", "Text", CellChangeTimes->{3.5359048507534103`*^9}]

but if I trig Ctrl+Shift+E and edit the CellExpression directly (eg. deleting the CellChangeTimes item, which should not affect the "appearance" of the cell), then convert the cell back to normal status, things get wrong again:

enter image description here

This kind of problem never happened when dealing with characters from CJK Unified Ideographs and CJK Unified Ideographs Extension A blocks, so I guess there may be a bug here.

So my questions is: While I do realize the "Unicode" encoding in Mathematica referring to only 16-bit Unicode while the CJK-B begin with x20000, is there any method to render and work with characters from CJK Unified Ideographs Extension B correctly?

Improve this question
$\endgroup$
  • $\begingroup$ I don't have time for this tonight, but I spent a lot of time with it before! Please also see the comments on this question and answer. You can also mail me, and I can perhaps send you some workaround code for handling them in the kernel, but I gave up on showing them in the front end. $\endgroup$ – Szabolcs Jan 18 '12 at 21:50
  • $\begingroup$ @Szabolcs Thanks a lot! I always wonder maybe at the very start Mathematica is not designed for dealing with full Unicode range. I'll check the link you provided and maybe mail you later. It's 6:00 in the morning in China and I have to sleep awhile now. $\endgroup$ – Silvia Jan 18 '12 at 22:03
  • 8
    $\begingroup$ Mathematica simply has no support for non-plane-0 characters. That it appears to temporarily work should not fool you into thinking that M-- knows anything about such values. Those who saw the R&D keynote at the 2011 Tech Conference may remember my relating the story of the pain we have experienced from the fact that we were extremely early adopters of Unicode, well before it was baked into OSes and the concept of Unicode planes had been fully developed. It affects every part of the system, and will be difficult and expensive to fix when we finally do fix it. $\endgroup$ – John Fultz Jan 30 '12 at 4:59
  • $\begingroup$ @JohnFultz There would be value in adding support to the kernel only first (without display support by the front end). By "support" I mean that string operations such as ToCharacter, FromCharacterCode, etc. should work correctly. $\endgroup$ – Szabolcs Mar 2 '12 at 14:27
  • $\begingroup$ @JohnFultz I noticed in 10.0 we can use the used-to-be-buggy characters now. May I consider it's a full support for all unicode characters? $\endgroup$ – Silvia Nov 18 '14 at 1:42
9
$\begingroup$

Reposting John Fultz’s comment above as a “community wiki” answer for everyone to improve:

Mathematica simply has no support for non-plane-0 characters. That it appears to temporarily work should not fool you into thinking that M-- knows anything about such values. Those who saw the R&D keynote at the 2011 Tech Conference may remember my relating the story of the pain we have experienced from the fact that we were extremely early adopters of Unicode, well before it was baked into OSes and the concept of Unicode planes had been fully developed. It affects every part of the system, and will be difficult and expensive to fix when we finally do fix it.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.