I have a large bzipped file that I want to read using OpenRead on a pipe with UTF8 encoding. Doing the following does not give me UTF-8 encoding. My $CharacterEncoding
is set to "UTF8"
.
f = OpenRead["!bunzip2 -c " <> filename]
OpenRead
does not have a CharacterEncoding
option like Import
does. So the following works:
txt = Import["!bunzip2 -c " <> filename, "Text", CharacterEncoding -> "UTF8"];
But Import
forces me to load all the data before operating on it which I don't want.
UPDATE1: For now I'm reading in the string as bytes and then converting to UTF8 manually
FromCharacterCode[ToCharacterCode[s], "UTF8"]
I notice Mathematica UI has problems printing certain Unicode characters as others stackoverflowers have noticed.
CharacterEncoding
is not supported for the low-level input streams whichOpenRead
works on. It is not in theOptions
. $\endgroup$ – librik Sep 8 '13 at 6:08