Change character encoding for file read using OpenRead

Question

I have a large bzipped file that I want to read using OpenRead on a pipe with UTF8 encoding. Doing the following does not give me UTF-8 encoding. My $CharacterEncoding is set to "UTF8".

f = OpenRead["!bunzip2 -c " <> filename]

OpenRead does not have a CharacterEncoding option like Import does. So the following works:

txt = Import["!bunzip2 -c " <> filename, "Text", CharacterEncoding -> "UTF8"];

But Import forces me to load all the data before operating on it which I don't want.

UPDATE1: For now I'm reading in the string as bytes and then converting to UTF8 manually

FromCharacterCode[ToCharacterCode[s], "UTF8"]

I notice Mathematica UI has problems printing certain Unicode characters as others stackoverflowers have noticed.

Answer 1

As OP has figured it out, the correct method for reading strings encoded in UTF-8 or similar and which contain special/accented characters is to read in raw bytes and convert them using FromCharacterCode. That is exactly what Import does under the hood (which is spelunkable if one dares to venture into the ~13500 lines of code associated with Import or the ~16000 lines of the Trace@Import[file, "Text"] stack).

Import first opens a stream and reads in the contents as Read[stream, Record, RecordSeparators -> {}], then processes the garbled string as:

 FromCharacterCode[ToCharacterCode["árvíztűrő tükörfúrÃ\.b3gép"], "UTF8"]

 "árvíztűrő tükörfúrógép"

Let's set up a file that contains UTF-8-encoded accented characters:

file = "test.txt";
stream = OpenWrite[file, CharacterEncoding -> "UTF-8"];
WriteString[stream, "árvíztűrő tükörfúrógép"];
Close@stream;

ReadList fails, Import succeeds but is slow, BinaryReadList is fast:

ReadList[file, String]
Import[file, "Text"]
FromCharacterCode[BinaryReadList@file, "UTF8"]

{"árvíztÅ\[PlusMinus]rő tükÃ\[Paragraph]rfúrÃ\.b3gép"}
"árvíztűrő tükörfúrógép"
"árvíztűrő tükörfúrógép"

Interestingly, the following encoding specification also works, though not included in $CharacterEncodings (internally, Import uses "UTF-8" instead of "UTF8" sometimes):

FromCharacterCode[BinaryReadList@file, "UTF-8"]

Answer 2

This is a guess, but a little exploration reveals a couple of things. First, if you look at the examples for the Method option for OpenRead you will notice that one of the stream methods is "File". This suggests you may be able to pass a CharacterEncoding to it, as follows:

strm = OpenRead["somefile", Method -> {"File", 
   CharacterEncoding -> "UTF8"}]

which you can check by looking at its Options:

Options[strm]
(* 
{
 BinaryFormat -> False, 
 Method -> {"File", 
  BinaryFormat -> False, 
  CharacterEncoding -> "UTF8"}
}
*)

So, it appears to have worked at least as far as the options are concerned. Now the "file" "!bunzip2 -c " <> filename is not a file, but is likely a "Pipe" as found by running

$InputStreamMethods
(* {"String", "HTTP", "WindowsTextFile", "GZIP", "Pipe", "File", "HTTP"} *)

So, I would try passing a CharacterEncoding in via the Method option.