# How do I save a Unicode string?

I am having a problem saving a long complicated string with Unicode characters in it. I have boiled the problem down, I think, to this minimal example:

hw = {72, 101, 108, 108, 111, 32, 8594, 32,
87, 111, 114, 108, 100, 33};
str = FromCharacterCode[hw]


which assigns a hello world string with a right arrow (the Unicode 8594 character) to str. If I then

ExportString[str, "Text"]


I get a different string, as the second line in

instead of the first line. I have a program that reads cells from a Notebook and custom converts it to a LaTeX string. I just cannot save the string. What am I doing wrong?

-
–  Mr.Wizard Jan 6 at 21:02
What is the logic of the junk ExportString prints into a cell? –  Hbar Jan 7 at 5:24

The correct character encoding for Export may help.

Export["hello.txt", str, "Text", CharacterEncoding -> "Unicode"]


does the trick for me:

-
Thanks. Did not know Export accepted character encodings. –  Hbar Jan 6 at 21:44

What is the logic of the junk ExportString prints into a cell?

On my system (Mathematica 7, Windows 7) I don't get quite the same result, but I assume the mechanism is similar. If we use ToCharacterCode to convert our intended string into UTF-8 we get multi-byte encoding:

utf8 = ToCharacterCode["Hello \[RightArrow] World!", "UTF8"]

{72, 101, 108, 108, 111, 32, 226, 134, 146, 32, 87, 111, 114, 108, 100, 33}


However when we convert this to a string we do not get the original:

FromCharacterCode[utf8]

"Hello â World!"


This is what I get when I use ExportString[str, "Text"] as shown in the question. I assume that a similar conversion is taking place.

-
I'm using MMA 9 on a Mac and this explains what I see. Thanks. –  Hbar Jan 7 at 7:27

If the goal is to write latex, why not use "Tex" export which gives nice looking Latex

hw = {72, 101, 108, 108, 111, 32, 8594, 32, 87, 111, 114, 108, 100, 33};
str = FromCharacterCode[hw];
ExportString[str, "Tex"]


or more simply

TeXForm[str]
\text{Hello $\rightarrow$ World!}

-
Thanks for the prompt answer. I would like to use the listings package for Input cells instead of line-by-line pmbs. Also, Mathematica notebooks have more structure than LaTeX, so they are easier to pre-process, allowing me to have cells marked latex, process grids as tabular when needed, ignore closed cells, etc. –  Hbar Jan 6 at 19:40
Sorry Hbar, but I have never used the listings package and don't know that pmbs is. I just was trying to give an easy way to "read from a Notebook and convert to a Latex string". –  bill s Jan 6 at 20:25
listings is a LaTeX package that helps one pretty-print programs and pmb (poor man's bold) is a macro from the TeXbook that creates bold characters by printing three slightly offset copies of its argument. –  Hbar Jan 7 at 5:29