I have a large bzipped file that I want to read using OpenRead on a pipe with UTF8 encoding. Doing the following does not give me UTF-8 encoding. My
$CharacterEncoding is set to
f = OpenRead["!bunzip2 -c " <> filename]
OpenRead does not have a
CharacterEncoding option like
Import does. So the following works:
txt = Import["!bunzip2 -c " <> filename, "Text", CharacterEncoding -> "UTF8"];
Import forces me to load all the data before operating on it which I don't want.
UPDATE1: For now I'm reading in the string as bytes and then converting to UTF8 manually
I notice Mathematica UI has problems printing certain Unicode characters as others stackoverflowers have noticed.