Skip to main content
26 votes
Accepted

Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails

I'm Riccardo, current developer of URLRead in WL and I have some experience working with encoding in WL. I would like to inform you that this is not a bug. In modern versions of mathematica we have ...
Riccardo Di Virgilio's user avatar
22 votes

Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails

Tracing the evaluation (Mathematica 11.1.1) shows that the string is passed to Developer`ReadRawJSONStream which actually produces the messages: ...
Alexey Popkov's user avatar
22 votes
Accepted

HMAC correct on 11.2 but wrong and different on 11.3 running same code

Indeed, Hash has changed for 11.3 in several ways. The one relevant here is that strings are now hashed according to their UTF-8 representation. As a simple ...
ilian's user avatar
  • 25.5k
14 votes
Accepted

What's purpose of function System`Private`LookupCodeByName and System`Private`LookupNameByCode

I tried random inputs until I got something to work. It looks like it has to do with character codes: System`Private`LookupNameByCode[200] ...
Greg Hurst's user avatar
  • 36.2k
12 votes
Accepted

Base64 Encoding and Decoding not correct?

The String format is binary, so there is no character encoding applied. If you use ISO8859-1 on https://www.base64decode.org, the output will match what you see in Mathematica. If you wish to Import ...
GenericAccountName's user avatar
12 votes
Accepted

Equivalent for \hookrightarrow, ↪

You can use an empty TemplateBox to create a new operator. For example: ...
Carl Woll's user avatar
  • 131k
12 votes

Does Mathematica have an equivalent to the Python 'translate' function?

Python String maketrans() Method Python code: ...
azerbajdzan's user avatar
  • 18.9k
11 votes
Accepted

Hexadecimal bytes as variable converted to binary bits

You cannot use the notation 16^^ with variable arguments, since it does not get "evaluated" like normal functions, but rather it is just a way of writing ...
Hausdorff's user avatar
  • 3,535
10 votes
Accepted

Fail to load a UTF8 package with one special Chinese character

Use Get["~/test.m", CharacterEncoding -> "UTF-8"] or change the value of $CharacterEncoding. Edit: Why the error message?...
Itai Seggev's user avatar
  • 14.2k
9 votes
Accepted

How to export strings as pure ASCII?

ExportAsASCII Currently I know only one well-documented and working (with exception for the NULL character \.00 and a few ...
Alexey Popkov's user avatar
9 votes
Accepted

Bug with Json payload with diacritics for HTTPRequest

ExportString JSON produces UTF8 encoded json string, if put as HTTPRequest body it will undergo another encoding unless you prevent it with an option: ...
Kuba's user avatar
  • 137k
9 votes
Accepted

Unicode symbols not encoded properly in BarcodeImage[] in Mathematica 13.3

I am not super comfortable with peculiarities of character encodings in Mathematica, but I would say this is a regression bug introduced when they changed the external library used for QR generation (...
Domen's user avatar
  • 26.5k
8 votes
Accepted

Find prefix of list in Association

If you are willing to use a simple string representation, then StringReplace might be adequate for your purpose: ...
WReach's user avatar
  • 69k
8 votes

Bug with Json payload with diacritics for HTTPRequest

I've had luck using the functions WriteRawJSONString and ReadRawJSONString from Developer ...
chuy's user avatar
  • 11.2k
8 votes
Accepted

Importing text generates whitespaces between every character

Your text file is saved with the byte order mark (BOM) (specifically the UTF-16 (LE) encoding). Use the following: ...
Domen's user avatar
  • 26.5k
8 votes
Accepted

Garbled code when using URLSubmit[] to request CJK web page

It appears that using BodyByteArray is the preferred approach. ...
Jean-Pierre's user avatar
  • 5,197
8 votes
Accepted

Does Mathematica have an equivalent to the Python 'translate' function?

...
Bob Hanlon's user avatar
  • 159k
8 votes

How can I export a text file in UTF-8 with a BOM (also known as UTF-8 with signature)?

I tried for a while, but I failed to find a built-in option to achieve this. (It's a bit strange to me, because the Import function clearly knows how to handle text ...
xzczd's user avatar
  • 66.9k
7 votes

How to workaround failures with Unicode filepaths?

Windows-specific: use short paths (short filenames, 8.3 file names) This method should work with all parts of the Mathematica system. Limitations short path can be found only for existing file/...
Alexey Popkov's user avatar
7 votes

Convert Unicode to Math Input

You can use RemoveDiacritics: RemoveDiacritics /@ {"a", "\:1d62"} {"a", "i"}
Greg Hurst's user avatar
  • 36.2k
7 votes

Unexpected behavior: missing characters in the code page 936

This can be fixed by substituting the encoding file or creating a new one. Import the definition of cp936: ...
rnotlnglgq's user avatar
  • 3,740
6 votes

Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails

Alexey Popkov's answer is correct. So as to who is to blame, it's Mathematica. The standard requires applications to accept all Unicode characters in strings. Quote from RFC 7159: Section 7: All ...
pj-yn9aq's user avatar
  • 161
6 votes

Import text outlines from PDF format properly

I reproduce the problem with versions 10.0.1, 10.4.1 and 11.2.0 on Windows 7 x64. The problem isn't related to the font used: the same happens when instead of ...
Alexey Popkov's user avatar
6 votes
Accepted

Exporting/Saving Notebook with Non English characters to PDF

I am on MMA 11.2 Win7-64. In order to reproduce the problem, I copied a few sentences from the Wikipedia page for MMA in Chinese (https://zh.wikipedia.org/wiki/Wolfram_Mathematica), then pasted them ...
MarcoB's user avatar
  • 67.3k
6 votes

Convert Unicode to Math Input

A very long answer for a very short question... A manually compiled list of Unicode subscript and superscript characters (from this post): ...
kglr's user avatar
  • 397k
6 votes

How do I add EOF character to a CSV file?

Adding EOF When Creating The File If we are creating the CSV file ourselves then we can open a stream, export the CSV, and write the EOF character separately at the end: ...
WReach's user avatar
  • 69k
6 votes
Accepted

One-hot vectors encoding with NetEncoder

The "Class" NetEncoder outputs one-hot vectors when the "UnitVector" form is specified: ...
Theo Tiger's user avatar
  • 1,273
6 votes

Convert integer to words of specific length

...
Syed's user avatar
  • 55.5k
5 votes

Find prefix of list in Association

This is probably not highly efficient but I think it does what you describe. ...
Mr.Wizard's user avatar
  • 272k
5 votes

Re-encode text file to use Mathematica's character name, like \[Alpha]

Here is a robust version of fromCode which uses only well-documented functionality, and correctly handles extended-ASCII and Unicode characters with which the ...
Alexey Popkov's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible