I have a large bzipped file that I want to read using OpenRead on a pipe with UTF8 encoding. Doing the following does not give me UTF-8 encoding. My $CharacterEncoding is set to "UTF8".

f = OpenRead["!bunzip2 -c " <> filename]

OpenRead does not have a CharacterEncoding option like Import does. So the following works:

txt = Import["!bunzip2 -c " <> filename, "Text", CharacterEncoding -> "UTF8"];

But Import forces me to load all the data before operating on it which I don't want.

UPDATE1: For now I'm reading in the string as bytes and then converting to UTF8 manually

FromCharacterCode[ToCharacterCode[s], "UTF8"]

I notice Mathematica UI has problems printing certain Unicode characters as others stackoverflowers have noticed.

  • $\begingroup$ CharacterEncoding is not supported for the low-level input streams which OpenRead works on. It is not in the Options. $\endgroup$ – librik Sep 8 '13 at 6:08

As OP has figured it out, the correct method for reading strings encoded in UTF-8 or similar and which contain special/accented characters is to read in raw bytes and convert them using FromCharacterCode. That is exactly what Import does under the hood (which is spelunkable if one dares to venture into the ~13500 lines of code associated with Import or the ~16000 lines of the Trace@Import[file, "Text"] stack).

Import first opens a stream and reads in the contents as Read[stream, Record, RecordSeparators -> {}], then processes the garbled string as:

 FromCharacterCode[ToCharacterCode["árvíztűrő tükörfúrÃ\.b3gép"], "UTF8"]
 "árvíztűrő tükörfúrógép"

Let's set up a file that contains UTF-8-encoded accented characters:

file = "test.txt";
stream = OpenWrite[file, CharacterEncoding -> "UTF-8"];
WriteString[stream, "árvíztűrő tükörfúrógép"];

ReadList fails, Import succeeds but is slow, BinaryReadList is fast:

ReadList[file, String]
Import[file, "Text"]
FromCharacterCode[BinaryReadList@file, "UTF8"]
{"árvíztÅ\[PlusMinus]rő tükÃ\[Paragraph]rfúrÃ\.b3gép"}
"árvíztűrő tükörfúrógép"
"árvíztűrő tükörfúrógép"

Interestingly, the following encoding specification also works, though not included in $CharacterEncodings (internally, Import uses "UTF-8" instead of "UTF8" sometimes):

FromCharacterCode[BinaryReadList@file, "UTF-8"]
| improve this answer | |

This is a guess, but a little exploration reveals a couple of things. First, if you look at the examples for the Method option for OpenRead you will notice that one of the stream methods is "File". This suggests you may be able to pass a CharacterEncoding to it, as follows:

strm = OpenRead["somefile", Method -> {"File", 
   CharacterEncoding -> "UTF8"}]

which you can check by looking at its Options:

 BinaryFormat -> False, 
 Method -> {"File", 
  BinaryFormat -> False, 
  CharacterEncoding -> "UTF8"}

So, it appears to have worked at least as far as the options are concerned. Now the "file" "!bunzip2 -c " <> filename is not a file, but is likely a "Pipe" as found by running

(* {"String", "HTTP", "WindowsTextFile", "GZIP", "Pipe", "File", "HTTP"} *)

So, I would try passing a CharacterEncoding in via the Method option.

| improve this answer | |
  • $\begingroup$ Unfortunately adding Method -> {"File", BinaryFormat -> False, CharacterEncoding -> "UTF8"} to OpenRead has no effect besides changing the result of Options[] call for the stream. Maybe this is a bug? I've put up my test file here. There are some 2-byte UTF-8 characters there. $\endgroup$ – Kartik Aug 24 '13 at 5:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.