27
votes
Accepted
Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails
I'm Riccardo, current developer of URLRead in WL and I have some experience working with encoding in WL.
I would like to inform you that this is not a bug.
In modern versions of mathematica we have ...
23
votes
Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails
Tracing the evaluation (Mathematica 11.1.1) shows that the string is passed to Developer`ReadRawJSONStream which actually produces the messages:
...
- 62.3k
22
votes
Accepted
HMAC correct on 11.2 but wrong and different on 11.3 running same code
Indeed, Hash has changed for 11.3 in several ways.
The one relevant here is that strings are now hashed according to their UTF-8 representation.
As a simple ...
- 25.5k
14
votes
Accepted
What's purpose of function System`Private`LookupCodeByName and System`Private`LookupNameByCode
I tried random inputs until I got something to work. It looks like it has to do with character codes:
System`Private`LookupNameByCode[200]
...
- 36.8k
12
votes
Accepted
Base64 Encoding and Decoding not correct?
The String format is binary, so there is no character encoding applied. If you use ISO8859-1 on https://www.base64decode.org, the output will match what you see in Mathematica.
If you wish to Import ...
- 1,593
12
votes
Accepted
Equivalent for \hookrightarrow, ↪
You can use an empty TemplateBox to create a new operator. For example:
...
- 132k
12
votes
Does Mathematica have an equivalent to the Python 'translate' function?
Python String maketrans() Method
Python code:
...
- 25.1k
11
votes
Accepted
Hexadecimal bytes as variable converted to binary bits
You cannot use the notation 16^^ with variable arguments, since it does not get "evaluated" like normal functions, but rather it is just a way of writing ...
- 3,535
10
votes
Accepted
Fail to load a UTF8 package with one special Chinese character
Use
Get["~/test.m", CharacterEncoding -> "UTF-8"]
or change the value of $CharacterEncoding.
Edit: Why the error message?...
- 14.3k
9
votes
Accepted
How to export strings as pure ASCII?
ExportAsASCII
Currently I know only one well-documented and working (with exception for the NULL character \.00 and a few ...
- 62.3k
9
votes
Accepted
Bug with Json payload with diacritics for HTTPRequest
ExportString JSON produces UTF8 encoded json string, if put as HTTPRequest body it will undergo another encoding unless you prevent it with an option:
...
- 138k
9
votes
Accepted
Unicode symbols not encoded properly in BarcodeImage[] in Mathematica 13.3
I am not super comfortable with peculiarities of character encodings in Mathematica, but I would say this is a regression bug introduced when they changed the external library used for QR generation (...
- 33.4k
8
votes
Accepted
Find prefix of list in Association
If you are willing to use a simple string representation, then StringReplace might be adequate for your purpose:
...
- 69.3k
8
votes
Bug with Json payload with diacritics for HTTPRequest
I've had luck using the functions WriteRawJSONString and ReadRawJSONString from Developer ...
- 11.2k
8
votes
Accepted
Importing text generates whitespaces between every character
Your text file is saved with the byte order mark (BOM) (specifically the UTF-16 (LE) encoding).
Use the following:
...
- 33.4k
8
votes
Accepted
Garbled code when using URLSubmit[] to request CJK web page
It appears that using BodyByteArray is the preferred approach.
...
- 5,222
8
votes
Accepted
8
votes
How can I export a text file in UTF-8 with a BOM (also known as UTF-8 with signature)?
I tried for a while, but I failed to find a built-in option to achieve this. (It's a bit strange to me, because the Import function clearly knows how to handle text ...
- 68.4k
7
votes
How to workaround failures with Unicode filepaths?
Windows-specific: use short paths (short filenames, 8.3 file names)
This method should work with all parts of the Mathematica system.
Limitations
short path can be found only for existing file/...
- 62.3k
7
votes
Convert Unicode to Math Input
You can use RemoveDiacritics:
RemoveDiacritics /@ {"a", "\:1d62"}
{"a", "i"}
- 36.8k
7
votes
Unexpected behavior: missing characters in the code page 936
This can be fixed by substituting the encoding file or creating a new one.
Import the definition of cp936:
...
- 3,750
6
votes
Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails
Alexey Popkov's answer is correct. So as to who is to blame, it's Mathematica. The standard requires applications to accept all Unicode characters in strings.
Quote from RFC 7159:
Section 7:
All ...
- 161
6
votes
Import text outlines from PDF format properly
I reproduce the problem with versions 10.0.1, 10.4.1 and 11.2.0 on Windows 7 x64. The problem isn't related to the font used: the same happens when instead of ...
- 62.3k
6
votes
Accepted
Exporting/Saving Notebook with Non English characters to PDF
I am on MMA 11.2 Win7-64. In order to reproduce the problem, I copied a few sentences from the Wikipedia page for MMA in Chinese (https://zh.wikipedia.org/wiki/Wolfram_Mathematica), then pasted them ...
- 67.7k
6
votes
Convert Unicode to Math Input
A very long answer for a very short question...
A manually compiled list of Unicode subscript and superscript characters (from this post):
...
- 401k
6
votes
How do I add EOF character to a CSV file?
Adding EOF When Creating The File
If we are creating the CSV file ourselves then we can open a stream, export the CSV, and write the EOF character separately at the end:
...
- 69.3k
6
votes
Accepted
One-hot vectors encoding with NetEncoder
The "Class" NetEncoder outputs one-hot vectors when the "UnitVector" form is specified:
...
- 1,273
6
votes
5
votes
Find prefix of list in Association
This is probably not highly efficient but I think it does what you describe.
...
- 273k
5
votes
Accepted
Only top scored, non community-wiki answers of a minimum length are eligible