3
$\begingroup$

I have a TSV file with string data in it that I want to import. While the TSV format is correctly parsed into a Mathematica table format using Import[<file>,"Table"], the actual text content is oddly formated.

A minimum example that reproduces the problem is this:

I have a text file called text.txt. Its only content is test.

When I use either of the commands ReadList["text.txt"] or Import["text.txt"] I obtain as output:

ÿþt e s t (visually)

When I copy paste the above visual output into a text editor, it reads as:

"ÿþt\.00e\.00s\.00t\.00"

How is it possible to obtain the expected output: test ?

This is Mathematica 11.2 running on Win 10.

$\endgroup$

1 Answer 1

5
$\begingroup$

Your text file is saved with the byte order mark (BOM) (specifically the UTF-16 (LE) encoding).

Use the following:

Import["test.txt", CharacterEncoding -> "Unicode"]
(* test *)

or change the encoding in your text file (for example with Notepad++) to UTF-8:

enter image description here

$\endgroup$
4
  • $\begingroup$ Thanks. It is strange that I have never encountered this issue in 10+ years of Mathematica usage on multiple OS and computers. $\endgroup$
    – tobalt
    Jan 3 at 12:01
  • $\begingroup$ How did you generate your text file? With Notepad on Win10? Be careful on the selected encoding in the Save dialog. $\endgroup$
    – Domen
    Jan 3 at 12:05
  • $\begingroup$ The file was generated from another tool, so this can be the origin. However, I then opened the file with Notepad (normal version) and edited it to contain test. And saved it under a new name. Seems that Notepad still preserved the original encoding of that other software. When importing and saving with Excel, Excel destroys the original encoding and removes the Mathematica import issues. However, your solution with CharacterEncoding -> "Unicode" is more elegant because I can import directly from the other software. $\endgroup$
    – tobalt
    Jan 3 at 12:08
  • 1
    $\begingroup$ @tobalt Serves you right! For using other tools, not just Mathematica... $\endgroup$ Jan 3 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.