4
$\begingroup$

I have a TSV file with string data in it that I want to import. While the TSV format is correctly parsed into a Mathematica table format using Import[<file>,"Table"], the actual text content is oddly formated.

A minimum example that reproduces the problem is this:

I have a text file called text.txt. Its only content is test.

When I use either of the commands ReadList["text.txt"] or Import["text.txt"] I obtain as output:

ÿþt e s t (visually)

When I copy paste the above visual output into a text editor, it reads as:

"ÿþt\.00e\.00s\.00t\.00"

How is it possible to obtain the expected output: test ?

This is Mathematica 11.2 running on Win 10.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
8
$\begingroup$

Your text file is saved with the byte order mark (BOM) (specifically the UTF-16 (LE) encoding).

Use the following:

Import["test.txt", CharacterEncoding -> "Unicode"]
(* test *)

or change the encoding in your text file (for example with Notepad++) to UTF-8:

enter image description here

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ Thanks. It is strange that I have never encountered this issue in 10+ years of Mathematica usage on multiple OS and computers. $\endgroup$
    – tobalt
    Commented Jan 3, 2022 at 12:01
  • $\begingroup$ How did you generate your text file? With Notepad on Win10? Be careful on the selected encoding in the Save dialog. $\endgroup$
    – Domen
    Commented Jan 3, 2022 at 12:05
  • $\begingroup$ The file was generated from another tool, so this can be the origin. However, I then opened the file with Notepad (normal version) and edited it to contain test. And saved it under a new name. Seems that Notepad still preserved the original encoding of that other software. When importing and saving with Excel, Excel destroys the original encoding and removes the Mathematica import issues. However, your solution with CharacterEncoding -> "Unicode" is more elegant because I can import directly from the other software. $\endgroup$
    – tobalt
    Commented Jan 3, 2022 at 12:08
  • 1
    $\begingroup$ @tobalt Serves you right! For using other tools, not just Mathematica... $\endgroup$
    – Anton Antonov
    Commented Jan 3, 2022 at 12:24
  • 1
    $\begingroup$ @rhemans, I don't know, but it is explicitly mentioned in the documentation for importing Text, together with the remark about the byte order mark. $\endgroup$
    – Domen
    Commented Feb 21, 2023 at 12:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.