Questions tagged [character-encoding]
Questions about character encoding and decoding. This includes, but is not restricted to, conversion of characters to their numerical value in a character table and vice versa and conversion of characters to strings and vice versa.
124
questions
67
votes
10
answers
5k
views
How to "Copy as Unicode" from a Notebook?
I spent some time manually editing a post replacing Mathematica ASCII \[Alpha] with Unicode α. I did this by laboriously ...
- 273k
34
votes
3
answers
10k
views
Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails
Bug introduced in 11.0 or earlier and persists through 12.0
(confirmed by WRI Support but read summary below)
Answers summary
I think we need a quick summary here to justify keeping bug.
Original ...
- 137k
20
votes
3
answers
1k
views
Unicode-aware string functions?
I've recently discovered that ToUpperCase is quite unreliable on non-ASCII input:
...
- 10.8k
20
votes
1
answer
1k
views
How to work with characters from CJK Unified Ideographs Extension B correctly?
Background:
Recently I'm trying to deal with some Chinese characters which belongs to the CJK Unified Ideographs Extension B block. The motive comes from some further interest following a former ...
- 27.6k
19
votes
1
answer
557
views
HMAC correct on 11.2 but wrong and different on 11.3 running same code
If you run this code on 11.2
...
- 1,299
19
votes
1
answer
785
views
How to add named characters to Mathematica 10 or later?
Any news regarding this for Mathematica 12?
It appears that native support for the Klingon alphabet has been removed from version 10:
\[KlingonA]
Syntax::...
- 273k
17
votes
3
answers
2k
views
How to workaround failures with Unicode filepaths?
(Cross-posted on the Wolfram Community, reported to the support as [CASE:3965891].)
The Mathematica's Kernel and FrontEnd currently work well with Unicode file/directory paths, but some other ...
- 62.1k
14
votes
2
answers
2k
views
Light syntax parser for Mathematica?
My netbook is too slow to install Mathematica. I'm not set against working in a text editor, but I like having an automated highlighting/formatting environment; can anyone recommend a good gedit ...
- 343
13
votes
2
answers
2k
views
Problem importing URL with Greek characters
I'm considering buying a car. So I thought why not make a web-crawler in Mathematica to pile-up car data? Brilliant idea.
Then I found this Greek website, gocar.gr, which just so happens to have all ...
- 343
13
votes
1
answer
537
views
Bug in handling backslashed Cyrillic symbols in v.10.0.0?
Bug introduced in 10.0.0 and fixed in 10.1
UPDATE: Now both the inputs "\б" and "\\б" are interpreted as ...
- 62.1k
13
votes
2
answers
869
views
Copying Greek text from notebooks as unicode [duplicate]
Possible Duplicate:
How to “Copy as Unicode” from a Notebook?
How can I copy Greek text from notebooks as proper Unicode that can be pasted into other applications? If I type some ...
- 236k
12
votes
3
answers
873
views
Bug with Json payload with diacritics for HTTPRequest
Working in the creations of APIs with Mathematica 12.2, I got this problem with diacritics when sending JSON data.
Here is a toy code using httpbin.
...
- 26.3k
12
votes
2
answers
558
views
How to generate characters by a function which works like the Esc + a + ESC
How to generate characters by a function which works like the Esc + a + ESC
...
- 8,629
12
votes
1
answer
1k
views
Exporting plots with Cyrillic text elements to pdf
I'm trying to export a plot with some added notes which happen to be in Russian.
Using Mathematica 8.04 and WinXP I evaluate
...
- 2,016
12
votes
1
answer
860
views
Exporting German Umlauts to PDF and importing them doesn't work - character encoding problem
I am using Mathematica 9.0.0.0 under German Windows 32bit. I am on trying to export and import German text to PDF and re-import it. It seems that there is a confusion about the character encoding. It ...
- 191
10
votes
1
answer
1k
views
Importing a Base64 encoded string
I'm handling some paragraphs encoded with Base64 from CSV file format. The problem is that Mathematica can't decode it. I'm thinking to write my decode method by my own, but I ask it first.
By ...
- 1,057
10
votes
1
answer
8k
views
Base64 Encoding and Decoding not correct?
When comparing the output of decoding a base64 encoded string, the Mathematica output and the output of https://www.base64decode.org does not match.
...
- 1,299
10
votes
1
answer
2k
views
How to export strings as pure ASCII?
The "PrintableASCII" characters set consists of 97 characters, including space (\[RawSpace]) and two control characters: the ...
- 62.1k
10
votes
1
answer
370
views
Defining my own invisible operator
There are some invisible operators in Mathematica (\[ImplicitPlus],\[InvisibleTimes],...
- 7,941
10
votes
1
answer
531
views
Converting text from unsupported encodings like 866 into known by Mathematica?
The list of character encodings currently supported by Mathematica is still extremely limited. For example, it supports only three standard IBM PC (OEM) code pages:
...
- 62.1k
9
votes
2
answers
826
views
Does Mathematica have an equivalent to the Python 'translate' function?
In Python you can make a character translation table with 'maketrans', passing it two strings of the same length. Then you can call 'translate' passing a new string and the table you just created. The ...
- 189
9
votes
2
answers
1k
views
Convert quoted printable aka PrintableASCII characters to UTF-8
I have several .eml email files to import. Some of them were saved as quoted-printable 7-bit encoded.
I skipped through the Mathematica documentation and found ...
- 353
9
votes
2
answers
1k
views
Hexadecimal bytes as variable converted to binary bits
I am working with hexadecimal bytes but need to shift them into lists of binary bits for a few operations and then back again.
For example, I would like C12B to turn into {1,1,0,0,0,0,0,1,0,0,1,0,1,0,...
- 193
8
votes
3
answers
320
views
Re-encode text file to use Mathematica's character name, like \[Alpha]
How can I re-encode a text file (or string) so that non-ASCII characters will be represented using Mathematica's standard names?
For example, if the original contents of the file was
...
- 236k
8
votes
1
answer
446
views
Exporting Cyrillics in PDF on Linux
I'm using Mathematica from Ubuntu. I need a normal font to display cyrillics inside PDFs. Any default font is very ugly, letters are too close together:
...
- 7,941
8
votes
2
answers
573
views
How do we turn off automatic smart quotes/special character substitution?
In Mathematica 8, whenever I enter "hello" into a cell of style "text", the quote marks are automatically converted to special characters:
...
- 835
8
votes
1
answer
278
views
Unicode symbols not encoded properly in BarcodeImage[] in Mathematica 13.3
Bug introduced in 13.2 or 13.3 and fixed in 14.1
Trying to generate a simple QR code with ← or U+2190. I got varying results: the same in Mathematica versions 12.3 and 13.1 but very different in ...
- 287
8
votes
1
answer
132
views
How to transform traditional Chinese string into simplified Chinese string in Mathematica?
There are two type of written Characters in Chinese: traditional Chinese and simplified Chinese. Is there a way to transform traditional Chinese string into simplified Chinese string in Mathematica? ...
- 17.3k
8
votes
0
answers
180
views
How to correctly CloudDeploy code which contains non-English characters?
As an illustration：
CloudDeploy[{"中文"}]
will give a url. After open it, I got following on the web page
{\:4e2d\:6587}
...
- 17.3k
7
votes
2
answers
412
views
Can I read Chinese characters with ReadList correctly?
Just try this example:
Export["a.txt", "这乱码问题该怎么解决呢\n***\n怎么解决嘛"]
ReadList["a.txt", Record]
...
- 67.7k
7
votes
1
answer
305
views
Equivalent for \hookrightarrow, ↪
Of course I can simply type the unicode ↪, but is there a operator with no built in meaning that renders like this?
The documentation guide/ArrowsAndArrowLikeForms presumably lists all arrows. If it ...
- 3,986
7
votes
1
answer
223
views
Fail to load a UTF8 package with one special Chinese character
Please check out the following code.
BeginPackage["WTF`"]
f::usage="blah"
Begin["`Private`"]
f[x_]:=1(*盘*);
End[]
EndPackage[]
I've save the code to ...
- 2,814
7
votes
1
answer
927
views
Export and Import with special unicode characters
I exported a list of characters say:
ClearAll[t, r, \[Theta], \[Phi]];
Export["hello.txt", {t, r, \[Theta], \[Phi]}, "List"];
the file has the correct symbols ...
- 517
7
votes
1
answer
382
views
How can I export a text file in UTF-8 with a BOM (also known as UTF-8 with signature)?
The default setting "UTF8"/"UTF-8" results in a text file without a byte order mark (BOM)/signature. What if ...
- 67.7k
7
votes
2
answers
2k
views
Change character encoding for file read using OpenRead
I have a large bzipped file that I want to read using OpenRead on a pipe with UTF8 encoding. Doing the following does not give me UTF-8 encoding. My ...
- 105
7
votes
1
answer
190
views
Unexpected behavior: missing characters in the code page 936
Bug persisting through 13.1.0
In the CP936.m file which will be loaded by
Internal`LoadEncodingFile@"CP936";
there ...
- 3,750
7
votes
1
answer
341
views
How to create a prefix code?
I am trying to create a demonstration of shannon code in mathematica.
Given a list of lengths (l1, l2,..lk) I need to create a list of binary lists (or list of ...
- 215
6
votes
2
answers
314
views
HTMLEncode a string for webMathematica
Is there a native Mathematica way to convert a string like "<select></select>" into ...
- 825
6
votes
5
answers
1k
views
Converting set of characters into numbers using list of replacement rules
I am trying to convert a text, as an example "aaaa" into a set of numbers as defined in a list of replacement rules, for example letterstonum . letterstonum takes in letters a to z and returns the ...
6
votes
1
answer
234
views
Garbled code when using URLSubmit[] to request CJK web page
When importing content containing Chinese (CJK) characters, correct results can be obtained by using URLread
...
- 185
6
votes
1
answer
2k
views
Sorting an Array with words in different languages
Array consists of words of different languages. I need to sort elements in reverse alphabetical order and make a list of elements, which consist of english letters.
Example of array which consist of ...
- 63
6
votes
1
answer
157
views
Exporting/Saving Notebook with Non English characters to PDF
On a related question, I now have another problem with saving notebook to PDF.
OS: Windows 7 SP1 English Ultimate
MMA: 11.3 X64
The Chinese characters displays ...
- 4,996
6
votes
1
answer
141
views
Encoding strings to workaround CloudDeployment bugs
Bug introduced with FormFunction and persisting through Cloud V1.48
CASE:4026150 confirmed
There are few problems when deploying simple ...
- 137k
6
votes
0
answers
423
views
Passing UTF-8 to wolframscript on the command line
I am trying to use Unicode input to wolframscript -c, like so:
$ ./wolframscript -c '1/∞'
â^(-1)
The source of this behavior ...
- 236
5
votes
1
answer
295
views
What's purpose of function System`Private`LookupCodeByName and System`Private`LookupNameByCode [closed]
These are useful function from their name.And they are kernel function since:
...
- 27.1k
5
votes
2
answers
277
views
Convert Unicode to Math Input
Any Ideas how I could convert unicode to Mathematica Input ?
For instance : I have the following Unicode: test= {"a", "\\:1d62"}
So here I would like to convert ...
- 3,093
5
votes
1
answer
368
views
A bug in converting some Unicode characters into FullForm?
Currently I know only one well-documented way to convert all special characters in an arbitrary string into their FullForm:
In ...
- 62.1k
5
votes
1
answer
113
views
Converting Unicode Number Subscripts to Standard Mathematica Subscript Notation
I am importing data
data={"t*α₁","α₁*α₄"}
But unfortunately, all of the subscripts are in Unicode:
...
- 267
5
votes
2
answers
273
views
Possible bug in Export with CharacterEncoding -> "PrintableASCII"
In the Documentation we read (emphasis is mine):
By default, the Wolfram System uses the character encoding "PrintableASCII" when saving notebooks and packages. ...
- 62.1k
5
votes
1
answer
507
views
Decoding GZIP encoded Body, BodyBytes (ByteArray) and BodyBytesArray from URLRead
When calling the Stack Exchange API v2.2 using the new (Mma11) function URLRead we get a GZIP encoded Body in alternative formats: ...
- 37k