# How to convert Strings from Mathematica (UTF-8) to Linux with locale "de_DE.UTF-8"?

I want to do file system operations with a Mathematica program. I see, that Mathematica and my Linux operating system use different encodings.

According to the unix command locale, I have LC_CTYPE="de_DE.UTF-8".

According to Mathematica, $CharacterEncoding returns me "UTF-8". There are several files and directories in my file system which have names using Umlaute, accented characters and the like. Rather than changing things in the existing file system and names in it, I want Mathematica to do the proper conversions when I send my commands to the OS using Run[command], where command is a string constructed in Mathematica which also uses of existing pathnames. What is the right conversion function to use in order to convert Strings which look reasonable in Mathematica such that, when put through Run[....] to the OS they become right in my locale. Currently I see strange letters which stem from the mismatch of character representation. ($Version is 12.1.1 for Linux x86 (64-bit) (June 19, 2020))

• Have you tried forcing a string conversion to UTF-8 with ToString[mystring, CharacterEncoding -> "UTF8"]? For example, ToString["Hanßen", CharacterEncoding -> "UTF8"] gives "HanÃŸen", correctly representing Unicode code point \:00df (Eszett) by two bytes. May 25 at 15:24
• Roman, interesting things happen if I do this: ow = OpenWrite["~/Hanßen.txt"]; WriteString[ow, ToString["Hanßen\n", CharacterEncoding -> "UTF8"]]; Close[ow];. In the Linux system, a file "Hanßen.txt" is created. But or = OpenRead["~/Hanßen.txt"]; string = string = Read[or, String]; Close[or]; string returns "HanA~\\237en". So different thing happen with the file's name and its content! What is the right Option to let the file's content be encoded like the file's name and how to let file contents appear in Mma as they appear in other programs? May 26 at 14:10
• Please try Export["~/Hanßen.txt", "Hanßen\n", "Text", CharacterEncoding -> "UTF8"] followed by Import["~/Hanßen.txt", "Text", CharacterEncoding -> "UTF8"]. Does this work better? See here May 26 at 20:16
• Roman, Yes, thank you: Export["~/Hanßen.txt", "some local characters \ aAáÁÂâäÄÀàeEéÉÊêëËÈèiIíÍÎîïÏÌìoOóÓÔôöÖÒouUúÚÛûüÜÙùßæÆœŒçÇøØÿŸ\nbut \ there are more of them!\n", CharacterEncoding -> "UTF8"]; works as expected and Import["~/Hanßen.txt", "Text", CharacterEncoding -> "UTF8"] returns some local characters \ aAáÁÂâäÄÀàeEéÉÊêëËÈèiIíÍÎîïÏÌìoOóÓÔôöÖÒouUúÚÛûüÜÙùßæÆœŒçÇøØÿŸ but there are more of them!. You are using Options for Import and Export, but Options[Export] returns an empty set! What must I do to read or write line by line? May 27 at 8:21
• Good! I really don't know about line-by-line reading; maybe ask a separate question? Or try reading the whole file into a long string and then use StringSplit to split it at newlines: StringSplit["abc abc\ndef\nghi\tj", "\n"]. May 27 at 8:33

Meanwhile I got hints for both directions and I have made two functions:

(* ReadLn reads one line of UTF-8 coded text from the open
stream inFile.
Returns the string including the linefeed (if there is one).
The last line is returned without linefeed, if there is
none after it. All subsequent ReadLn return the
*)
Block[{buffer, by}                       (* Block ReadLn *)
, buffer = {};
While[True
If[ (by === EndOfFile)
, Break[]                         (* === Break ===> *)
, AppendTo[buffer, by];
If[ (by == 10),  Break[]]       (* === Break ===> *)
];
];
If[by === EndOfFile                   (* observe: test with === *)
, EndOfFile                          (* file is exhausted *)
, FromCharacterCode[buffer, "UTF8"]  (* this line is the clue! *)
]

(* WriteLn[outFile_OutputStream, string_String] writes the given
string and an appended linefeed to the open file handle outFile.
WriteLn converts the given string to UTF-8 which is the default
of StringToByteArray. - Since from the Unix side the file is
a text file but Mathematica does BinaryWrite only for binary
files, the write operation is wrapped with Off ... On for the
error which would be issued in such a situation.
*)
WriteLn[outFile_OutputStream, string_String] :=
(Off[BinaryWrite::bfmt];
BinaryWrite[outFile, StringToByteArray[string <> "\n"]];
On[BinaryWrite::bfmt]
);


For the write direction I finally found an even simpler solution using Export. According to help, in Export[dest, ...], dest can be a file name or a pipe, a local or a cloud object. An open stream is not mentioned there. However it works:

outFile = OpenWrite["u"];
Export[outFile
, "aAáÁÂâäÄÀàeEéÉÊêëËÈèiIíÍÎîïÏÌìoOóÓÔôöÖÒouUúÚÛûüÜÙùßæÆœŒçÇøØÿŸðÐẞĸµđŋ«¢»\n"
, "Text"
, CharacterEncoding -> "UTF8"
];
Export[outFile
, "Es ist schön, daß das auch geht."
, "Text"
, CharacterEncoding -> "UTF8"
];
Close[outFile];


Unfortunately, Import is not a true mirror image of Export:

1. An open stream as first argument is rejected. A file designation is required, as it would be used for OpenRead to make an input stream.
2. If the second parameter ("format") is set to "String", it can not be combined with CharacterEncoding->"UTF8".

However, among \$ImportFormats there is an item "Text" which can be combined with CharacterEncoding->"UTF8". But Import["1.txt", "Text", CharacterEncoding -> "UTF8"] returns the whole file content at once, which requires more memory in the Mathematica session.

The lower level functions Read and Write have no option CharacterEncoding->"UTF".

On my computer,

Timing[s = Import["r", "Text", CharacterEncoding -> "UTF8"]]


takes 1.40854 seconds for a 33,6MB text file consisting of 118114 lines.

The quickest way to split the string s into a list of strings is sp=StringSplit[s, RegularExpression["\\n"]]: in my example it takes about 0.1 seconds only. The time consumption for

Timing[While[(s = ReadLn[inFile]) =!= EndOfFile, Null]]


is 140 seconds. However, my first solution would still be applicable for input files 1000 times as big, the second one would probably let my computer swap forever.

The best thing would be, if WRI let CharacterEncoding->"UTF8" become options for Read and Write (an possibly all other input/output functions) in conjunction with "String".