Import create evil invisible characters when import txt including ellipsis:"..."

Question

I'm using Mathematica10.0.2 on Win7

suppose I have a txt file named test_Import_characters.txt(http://pan.baidu.com/s/1hsvy6xQ) which contains two lines of string example(copied from http://broadskyblog.blogspot.com/):

in the meantime
in the next few…+NOUN

If I copied string into Mathematica directly and FullForm it, it reads:

"in the meantime
 in the next few\[Ellipsis]+NOUN"

It's fine. However, if I Import it and then FullForm it, strange thing happened:

However, If evaluate

FullForm@Import@
Export["D:\\English\\fixed_expression\\test_Import_characters_2.
txt", "in the meantime
in the next few\[Ellipsis]+NOUN", "Text"]

then all thing is fine, no Dis-cre-tion-ary-Hyp-hen appeared.

the test_Import_characters.txt could be download from http://pan.baidu.com/s/1hsvy6xQ or you can copy it from http://broadskyblog.blogspot.com/ and paste it into a txt file.

Problem remains even Import the file with explicit options CharacterEncoding -> "UTF-8":

FullForm[Import[
"D:\\English\\fixed_expression\\test_Import_characters.txt", "Text",
CharacterEncoding -> "UTF-8"]]

Dis-cre-tion-ary-Hyp-hen still appeared

Answer 1

When working with text, you must be aware of character encodings.

Today it is best practice to always use UTF-8, which is an international standard and supported by all modern software, all over the world.

Be aware of what encoding your text editor uses, and try to set it to UTF-8.

What happened here is that you copied the text from the website, and pasted it into a text editor. For whatever reason (possibly operating system settings), that text editor saved it using the GBK encoding. This encoding is an old Chinese (i.e. not international) standard, meant specifically for Chinese text. Mathematica does not support GBK.

Solution:

Use UTF-8 and import as Import["file.txt", "Text"] or possibly Import["file.txt", "Text", CharacterEncoding -> "UTF-8"]