# Fail to load a UTF8 package with one special Chinese character

Please check out the following code.

BeginPackage["WTF"]
f::usage="blah"
Begin["Private"]
f[x_]:=1(*盘*);
End[]
EndPackage[]


I've save the code to WTF.wl in VS Code with UTF8 encoding and put the file into the directory $UserBaseDirectory\Applications. But I failed to load it. << WTF Syntax::sntue: Unexpected end of file (probably unfinished expression) (line 7 of "WTF").  However, if I delete the culprit (*盘*), or change it into another Chinese character, or move it before :=, or even append a space (*盘 *), then everything works. My personal guess is that there maybe bugs of dealing with the non-English characters, but I cannot figure out the reasons. Can someone explain this weird behavior? • Mathematica does not read packages as UTF8. It reads them as $CharacterEncoding, which may differ from computer to computer (and will typically differ between Linux/Mac and Windows). It is not safe to encode package source as UTF8. I understand that this is inconvenient, but unfortunately it seems to be the case. You could complain to Wolfram Support. – Szabolcs Jun 13 '18 at 18:42
• If you open the package file with the front end, and re-save it, it should transform non-ASCII characters to a portable representation. However, these representations won't be very usable with other text editors (so don't overwrite your work!) – Szabolcs Jun 13 '18 at 18:44
• @Szabolcs Thanks for your quick and helpful reply. Yes, I also work around this problem by opening and resaving in the frond-end, but this make the upload to GitHub much more frustrating. Hope WRI could offer a solution. – luyuwuli Jun 13 '18 at 18:48
• I am curious about your operating system, and the value of $CharacterEncoding in Mathematica on your system. What are they? – Szabolcs Jun 13 '18 at 19:50 • @Szabolcs Windows 10 Pro Chinese, $CharacterEncoding returns CP936, and I can indeed load the package after setting $CharacterEncoding="UTF8". Although this doesn't directly answer the question in the OP. Since I previously assumed the encoding issue only involves garbled not the error like this. – luyuwuli Jun 13 '18 at 20:03 ## 1 Answer Use Get["~/test.m", CharacterEncoding -> "UTF-8"]  or change the value of $CharacterEncoding.

Edit: Why the error message?

What you are seeing is not a bug. Both UTF-8 and CP936 are variable-length encodings. Therefore, the way characters are (mis)interpreted depends on their exact sequence. With your original comment, your character and the closing asterisk becomes to Chinese characters, so the comment is never closed:

FromCharacterCode[ToCharacterCode["(*盘*)", "UTF-8"], "CP936"]
(* "(*鐩頪)" *)


Hence, the file ends in the middle of a comment, and the error message is correct. If you add a space inside the comment, you now get a different Chinese character and an asterisk, so you have a valid comment:

FromCharacterCode[ToCharacterCode["(*盘 *)", "UTF-8"], "CP936"]
(* "(*鐩頠*)" *)


You now have a syntactically valid file and no error message.

• Thanks. I also overcome the problem by setting \$CharacterEncoding="UTF8"`. But what I really want to know is the reason why this special character in the comment causes an error, yet slightly modification of the comment in the code won't have this issue. – luyuwuli Jun 15 '18 at 16:08
• See my updated answer. – Itai Seggev Jun 15 '18 at 17:20
• Thanks for your explanations, now I get it. – luyuwuli Jun 17 '18 at 13:36