24 votes
Accepted

Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails

I'm Riccardo, current developer of URLRead in WL and I have some experience working with encoding in WL. I would like to inform you that this is not a bug. In modern versions of mathematica we have ...
22 votes
Accepted

HMAC correct on 11.2 but wrong and different on 11.3 running same code

Indeed, Hash has changed for 11.3 in several ways. The one relevant here is that strings are now hashed according to their UTF-8 representation. As a simple ...
  • 25.3k
21 votes

Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails

Tracing the evaluation (Mathematica 11.1.1) shows that the string is passed to Developer`ReadRawJSONStream which actually produces the messages: ...
14 votes
Accepted

What's purpose of function System`Private`LookupCodeByName and System`Private`LookupNameByCode

I tried random inputs until I got something to work. It looks like it has to do with character codes: System`Private`LookupNameByCode[200] ...
  • 34.5k
12 votes
Accepted

Base64 Encoding and Decoding not correct?

The String format is binary, so there is no character encoding applied. If you use ISO8859-1 on https://www.base64decode.org, the output will match what you see in Mathematica. If you wish to Import ...
12 votes
Accepted

Equivalent for \hookrightarrow, ↪

You can use an empty TemplateBox to create a new operator. For example: ...
  • 127k
12 votes

Does Mathematica have an equivalent to the Python 'translate' function?

Python String maketrans() Method Python code: ...
  • 5,915
11 votes
Accepted

Hexadecimal bytes as variable converted to binary bits

You cannot use the notation 16^^ with variable arguments, since it does not get "evaluated" like normal functions, but rather it is just a way of writing ...
  • 3,435
9 votes
Accepted

How to export strings as pure ASCII?

ExportAsASCII Currently I know only one well-documented and working (with exception for the NULL character \.00 and a few ...
9 votes
Accepted

Fail to load a UTF8 package with one special Chinese character

Use Get["~/test.m", CharacterEncoding -> "UTF-8"] or change the value of $CharacterEncoding. Edit: Why the error message?...
  • 13.9k
9 votes
Accepted

Bug with Json payload with diacritics for HTTPRequest

ExportString JSON produces UTF8 encoded json string, if put as HTTPRequest body it will undergo another encoding unless you prevent it with an option: ...
  • 135k
8 votes
Accepted

Find prefix of list in Association

If you are willing to use a simple string representation, then StringReplace might be adequate for your purpose: ...
  • 67.9k
8 votes

Bug with Json payload with diacritics for HTTPRequest

I've had luck using the functions WriteRawJSONString and ReadRawJSONString from Developer ...
  • 10.8k
8 votes
Accepted

Garbled code when using URLSubmit[] to request CJK web page

It appears that using BodyByteArray is the preferred approach. ...
  • 5,147
8 votes
Accepted

Does Mathematica have an equivalent to the Python 'translate' function?

...
  • 140k
7 votes

How to workaround failures with Unicode filepaths?

Windows-specific: use short paths (short filenames, 8.3 file names) This method should work with all parts of the Mathematica system. Limitations short path can be found only for existing file/...
7 votes

Convert Unicode to Math Input

You can use RemoveDiacritics: RemoveDiacritics /@ {"a", "\:1d62"} {"a", "i"}
  • 34.5k
7 votes

Unexpected behavior: missing characters in the code page 936

This can be fixed by substituting the encoding file or creating a new one. Import the definition of cp936: ...
  • 3,630
7 votes
Accepted

Importing text generates whitespaces between every character

Your text file is saved with the byte order mark (BOM) (specifically the UTF-16 (LE) encoding). Use the following: ...
  • 12.3k
6 votes
Accepted

Converting set of characters into numbers using list of replacement rules

After taking the string and making individual characters you can apply LetterNumber to each character. This can be done by Map. ...
  • 14.3k
6 votes

Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails

Alexey Popkov's answer is correct. So as to who is to blame, it's Mathematica. The standard requires applications to accept all Unicode characters in strings. Quote from RFC 7159: Section 7: All ...
  • 161
6 votes

Import text outlines from PDF format properly

I reproduce the problem with versions 10.0.1, 10.4.1 and 11.2.0 on Windows 7 x64. The problem isn't related to the font used: the same happens when instead of ...
6 votes
Accepted

Exporting/Saving Notebook with Non English characters to PDF

I am on MMA 11.2 Win7-64. In order to reproduce the problem, I copied a few sentences from the Wikipedia page for MMA in Chinese (https://zh.wikipedia.org/wiki/Wolfram_Mathematica), then pasted them ...
  • 64.1k
6 votes

Convert Unicode to Math Input

A very long answer for a very short question... A manually compiled list of Unicode subscript and superscript characters (from this post): ...
  • 356k
6 votes

How do I add EOF character to a CSV file?

Adding EOF When Creating The File If we are creating the CSV file ourselves then we can open a stream, export the CSV, and write the EOF character separately at the end: ...
  • 67.9k
6 votes
Accepted

One-hot vectors encoding with NetEncoder

The "Class" NetEncoder outputs one-hot vectors when the "UnitVector" form is specified: ...
  • 1,253
5 votes
Accepted

Encode accents for html

I think HTML export in Mathematica is based on standards that date back to way before anyone had even heard of HTML5. E.g., the Import docs for HTML say it ...
  • 96.2k
5 votes
Accepted

How to translate/convert UTF-16 code to its corresponding word/characters by Mathematica?

I'm going to assume you want this specific char: Unicode Han Character The issue is not with Mathematica's character code conversion, but instead the encoding source. The value you have obtained is ...
5 votes
Accepted

Re-encode text file to use Mathematica's character name, like \[Alpha]

Based off information from this thread, the following should work. ...
  • 34.5k
5 votes

Re-encode text file to use Mathematica's character name, like \[Alpha]

Here is a robust version of fromCode which uses only well-documented functionality, and correctly handles extended-ASCII and Unicode characters with which the ...

Only top scored, non community-wiki answers of a minimum length are eligible