# Import create evil invisible characters when import txt including ellipsis:“…”

I'm using Mathematica10.0.2 on Win7

suppose I have a txt file named test_Import_characters.txt(http://pan.baidu.com/s/1hsvy6xQ) which contains two lines of string example(copied from http://broadskyblog.blogspot.com/):

in the meantime
in the next few…+NOUN


If I copied string into Mathematica directly and FullForm it, it reads:

"in the meantime
in the next few\[Ellipsis]+NOUN"


It's fine. However, if I Import it and then FullForm it, strange thing happened:

However, If evaluate

FullForm@Import@
Export["D:\\English\\fixed_expression\\test_Import_characters_2.
txt", "in the meantime
in the next few\[Ellipsis]+NOUN", "Text"]


then all thing is fine, no Dis-cre-tion-ary-Hyp-hen appeared.

the test_Import_characters.txt could be download from http://pan.baidu.com/s/1hsvy6xQ or you can copy it from http://broadskyblog.blogspot.com/ and paste it into a txt file.

Problem remains even Import the file with explicit options CharacterEncoding -> "UTF-8":

FullForm[Import[
"D:\\English\\fixed_expression\\test_Import_characters.txt", "Text",
CharacterEncoding -> "UTF-8"]]


Dis-cre-tion-ary-Hyp-hen still appeared

• Import does import your text file as it is. It is probably an encoding issue. I cannot tell what encoding your file has as none that I tried gave the original from the website. Try creating a UTF-8 encoded file and import using Import["file.txt", "Text"]. – Szabolcs Jul 2 '16 at 12:45
• @Szabolcs Err..FullForm@Import@ Export["D:\\English\\fixed_expression\\test_Import_characters_7. txt", "in the meantime in the next few[Ellipsis]+NOUN", "Text", CharacterEncoding -> "UTF8"] didn't reproduce the Dis-cre-tion-ary-Hyp-hen character? – Harry Jul 2 '16 at 12:54

When working with text, you must be aware of character encodings.

Today it is best practice to always use UTF-8, which is an international standard and supported by all modern software, all over the world.

Be aware of what encoding your text editor uses, and try to set it to UTF-8.

What happened here is that you copied the text from the website, and pasted it into a text editor. For whatever reason (possibly operating system settings), that text editor saved it using the GBK encoding. This encoding is an old Chinese (i.e. not international) standard, meant specifically for Chinese text. Mathematica does not support GBK.

Solution:

Use UTF-8 and import as Import["file.txt", "Text"] or possibly Import["file.txt", "Text", CharacterEncoding -> "UTF-8"]

• Dis-cre-tion-ary-Hyp-hen still appeared if FullForm[Import[ "D:\\English\\fixed_expression\\test_Import_characters.txt", "Text", CharacterEncoding -> "UTF-8"]] – Harry Jul 2 '16 at 13:00
• @Harry There is no discretionary hyphen in the file you posted. It is being misinterpreted by the importer which guesses at the encoding. Convert the file to UTF8 first, then import a UTF8 encoded file with Mathematica. Your file is GBK encoded because you saved it like that (possibly unaware of what encoding was chosen). Mathematica does not support the GBK encoding (at least not on my platform) and cannot import this file. – Szabolcs Jul 2 '16 at 13:11
• @Harry There is no good reason to use GBK over Unicode today even for Chinese text and for text in other languages it seems like a rather bad idea. – Szabolcs Jul 2 '16 at 13:14
• Err.. How..how to convert a file to UTF8/different encoding form? – Harry Jul 2 '16 at 13:17
• @Harry Use a text editor which allows choosing the encoding when saving. Make sure that when you open your file, the editor interprets it using the correct encoding. You may need to explicitly choose GBK unless the editor chooses that by default on your system. It's up to you to find a text editor for this, any good programmer's editor can do it. I use Vim but I do not recommend it because it takes a while to learn it. – Szabolcs Jul 2 '16 at 13:24