7
$\begingroup$

Just try this example:

Export["a.txt", "这乱码问题该怎么解决呢\n***\n怎么解决嘛"]
ReadList["a.txt", Record]
{"è¿ä¹\[PlusMinus]ç é®é¢è¯¥æä¹è§£å\.b3å¢", "***", "æä¹è§£å\.b3å"}

I know Import can handle it correctly:

StringSplit@Import["a.txt"]
{"这乱码问题该怎么解决呢", "***", "怎么解决嘛"}

But ReadList is generally much faster, right?

Well, after scanning this post and the links inside it I noticed that Mathematica may be just not good at this, but still want to make sure if it's not possible.

$\endgroup$
9
  • 1
    $\begingroup$ "But ReadList is generally much faster, right?" Maybe not having to consider language encodings is one the reasons it is faster. $\endgroup$ – m_goldberg May 26 '14 at 10:13
  • 2
    $\begingroup$ Have a look at Trace[Import["a.txt"]] $\endgroup$ – mfvonh May 26 '14 at 16:29
  • 1
    $\begingroup$ ReadList does not currently support character encodings. You can use it to read the data as a sequence of pure bytes and then convert (using ToCharacterCode and FromCharacterCode), but you can't get it directly from ReadList. $\endgroup$ – librik May 27 '14 at 4:32
  • 1
    $\begingroup$ It's not just Chinese, it's anything non-Latin-1. Try Paul Erdős and the ő will get mangled. $\endgroup$ – Szabolcs May 29 '14 at 12:01
  • 1
    $\begingroup$ @matheorem How did you export the list? This is my timing: i.stack.imgur.com/6QQek.png $\endgroup$ – xzczd Aug 30 '16 at 11:39
4
$\begingroup$

I think the most straightforward method is to read the data as a byte content, then interpret that as an UTF8 text. This is what it would look like:

FromCharacterCode[BinaryReadList["a.txt"], "UTF8"]

It will be slightly more performant than the other suggestions as it avoids any unnecessary conversions. Be aware that you need to break into lines via e.g. StringSplit if so desired...

$\endgroup$
9
$\begingroup$

Following the comments above, I think I've managed to find the answer, that is, as m_goldberg and librik said, ReadList doesn't support character encoding, and maybe that's one of the reasons it's fast.

However, that doesn't mean we can't make use of ReadList. In fact, following the advice from mfvonh, I found that Import internally uses ReadList to read a.txt first and then converts it to the right encoding with ToCharacterCode and FromCharacterCode after a lot of judgments that I don't understand very well and seem to be redundant. So why not omit those judgments?:

Export["a.txt", "这乱码问题该怎么解决呢\n***\n1234\n这样解决呀"];
FromCharacterCode[ToCharacterCode[ReadList["a.txt", Record](*,"ISOLatin1"*)], 
                  "UTF8"] // AbsoluteTiming
Import["a.txt"]; // AbsoluteTiming
{0.0010000, {"这乱码问题该怎么解决呢", "***", "1234", "这样解决呀"}}
{0.0440000, Null}

Not sure if this will fail in more complicated cases.

$\endgroup$
1
  • $\begingroup$ 为这么萌的example点个赞 $\endgroup$ – Yi Wang Jul 17 '14 at 11:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.