# Tag Info

24

I'm Riccardo, current developer of URLRead in WL and I have some experience working with encoding in WL. I would like to inform you that this is not a bug. In modern versions of mathematica we have ByteArray, and this is a representation of bytes. But for decades strings have been both bytes and "unicode" at the same time. The problem here is that ...

23

Declaration: This method for Windows is based on the .NET code from Todd Gayley's this wonderful answer. My .NET knowledge is absolutely ZERO, all credit goes to Todd. Code: The main idea is to extract the "Input"-style code string, convert it to the UTF-16 little endian form, which is the standard byte order in Windows, feed the bytes to system ...

22

Indeed, Hash has changed for 11.3 in several ways. The one relevant here is that strings are now hashed according to their UTF-8 representation. As a simple example of the difference, consider the following string, which has only one character. c = FromCharacterCode[217]; (* "\[CapitalUGrave]" *) Its UTF-8 representation consists of the two (decimal) ...

21

Tracing the evaluation (Mathematica 11.1.1) shows that the string is passed to DeveloperReadRawJSONStream which actually produces the messages: DeveloperReadRawJSONStream[StringToStream@string, "IssueMessagesAs" -> Import] General::jsonoutofrangeunicode: Out of range unicode code point encountered. Import::jsoninvalidtoken: Invalid token ...

14

Here's how to add a permanent menu item and keyboard shortcut for Silvia's solution. (This can be easily adapted to Jens' solution for Mac OS.) 1. Add Package Copy Silvia's first code block that starts with Needs["NETLink"] to a file and save it as UniCodeCopy.m in one of the directories included in $Path. 2. Initialize the Front End Save the following ... 14 I tried random inputs until I got something to work. It looks like it has to do with character codes: SystemPrivateLookupNameByCode[200] "CapitalEGrave" And then we can plug in the reverse: SystemPrivateLookupCodeByName["CapitalEGrave"] 200 Edit by yode as J.M.'s comment Grid[DeleteCases[{#, FromCharacterCode[#], SystemPrivate... 13 That's because "Base64" is not a format but an encoding and you still need to tell Mathematica what format to import after decoding from Base64. This is described in the documentation. Try this: ImportString[s, {"Base64", "String"}] 12 For file operations two workarounds currently work: 1) Use slash / instead of double backslash in the paths to files under Windows: Import["D:/test/кириллическое название/файл.txt"] 2) Use FileNameJoin or StringJoin: Import[FileNameJoin[{"D:", "test", "кириллическое название", "файл.txt"}]] Import[StringJoin[{"D:", "\\", "test", "\\", "кириллическое ... 11 The different HTML entities are stored in SystemConvertMLStringDataDump$HTMLEntities on version 9 and from here, it's a simple StringReplace: StringReplace["<select></select>", SystemConvertMLStringDataDump$HTMLEntities] (* "&lt;select&gt;&lt;/select&gt;" *) 11 Here is a function that copies a Unicode string to the clipboard using JLink: Needs["JLink"]; InstallJava[]; LoadJavaClass["java.awt.Toolkit", AllowShortContext -> False]; uniclip[s_String] := JavaBlock[ javaawtToolkitgetDefaultToolkit[]@getSystemClipboard[]@setContents[#, #]& @ JavaNew["java.awt.datatransfer.StringSelection", s] ];... 11 The String format is binary, so there is no character encoding applied. If you use ISO8859-1 on https://www.base64decode.org, the output will match what you see in Mathematica. If you wish to Import as text with a character encoding, it can be done so like this: s="hqKtggfNEN7UOZG7rUlnIOsqDW+hstKUZ+3iOlltlDgE9GGZpLlvGAf/IJVUxAcNF3UBPvQa+GjGMeRdEFssng=="; ... 10 You cannot use the notation 16^^ with variable arguments, since it does not get "evaluated" like normal functions, but rather it is just a way of writing integers in different bases. For more information you can look at this question. A way to implement what you want would be hexToBinary[hexstring_String] := IntegerDigits[FromDigits[hexstring,16],2]... 9 Following the comments above, I think I've managed to find the answer, that is, as m_goldberg and librik said, ReadList doesn't support character encoding, and maybe that's one of the reasons it's fast. However, that doesn't mean we can't make use of ReadList. In fact, following the advice from mfvonh, I found that Import internally uses ReadList to read ... 9 Something like: ExportString[Cell[TextData["<select></select>"],"Text"],"HTML","FullDocument" -> False] produces: <p class="Text"> &lt;select&gt;&lt;/select&gt; </p> which might also be a good start. 9 ExportAsASCII Currently I know only one well-documented and working (with exception for a few characters from Unicode Private Use Area which normally shouldn't appear in code) method to convert all special and non-ASCII characters in an arbitrary string into "PrintableASCII" programmatically: toInputForm[s_String] := StringTake[ ToString[s, InputForm, ... 9 Use Get["~/test.m", CharacterEncoding -> "UTF-8"] or change the value of$CharacterEncoding. Edit: Why the error message? What you are seeing is not a bug. Both UTF-8 and CP936 are variable-length encodings. Therefore, the way characters are (mis)interpreted depends on their exact sequence. With your original comment, your character and the closing ...

9

ExportString JSON produces UTF8 encoded json string, if put as HTTPRequest body it will undergo another encoding unless you prevent it with an option: HTTPRequest[..., CharacterEncoding -> None] It is analogous to the double decoding issue which you address with bodybytes//FromCharacterCode//ImportString. Exchanging JSON via http requests This is very ...

8

If you are willing to use a simple string representation, then StringReplace might be adequate for your purpose: decode[s_] := StringReplace[s, {"1" -> "a", "01" -> "b", "00" -> "c"}] decode["110100"] (* "aabc" *) decode["101100101"] (* "abacab" *)

8

I've had luck using the functions WriteRawJSONString and ReadRawJSONString from Developer context. Check out: WriteRawJSONString vs ExportString? HTTPRequest[ "https://httpbin.org/anything", <|Method -> "POST", "Body" -> (<|"textWithDiacritics" -> "àéíóú"|> // Developer...

7

m = {"computer", "экзамен", "elephant", "стол", "bread", "телефон", "exception", "desktop", "best", "колонка", "zoom", "saphire", "ярость"}; First Sort it: sortedm = SortBy[m, First@ToCharacterCode@# &] {"best", "bread", "computer", "desktop", "elephant", "exception", "saphire", "zoom", "колонка", "стол", "телефон", "экзамен", "ярость"} Then ...

7

Windows-specific: use short paths (short filenames, 8.3 file names) This method should work with all parts of the Mathematica system. Limitations short path can be found only for existing file/directory short paths exist only if generation of them isn't disabled by administrator (by default it is enabled) may not work for network paths this method isn't ...

7

You can use RemoveDiacritics: RemoveDiacritics /@ {"a", "\:1d62"} {"a", "i"}

6

After taking the string and making individual characters you can apply LetterNumber to each character. This can be done by Map. Map[LetterNumber, Characters["aaaaa"]] produces {1, 1, 1, 1, 1}

6

This is not a complete answer. However, since no other reply is forthcoming here are the lines that may be appended to UnicodeCharacters.tr as mentioned in the question. This at least allows these characters to be recognized by the Front End rather than producing a syntax error. The glyphs remain broken. 0xF8D0 \[KlingonA] () Letter 0xF8D1 ...

6

I am on MMA 11.2 Win7-64. In order to reproduce the problem, I copied a few sentences from the Wikipedia page for MMA in Chinese (https://zh.wikipedia.org/wiki/Wolfram_Mathematica), then pasted them into MMA and changed the cell to text. Trying to save as PDF from there produced the garbage you saw. However, I found the problem to be the font that is ...

6

A very long answer for a very short question... A manually compiled list of Unicode subscript and superscript characters (from this post): unicodeSubSuperscriptList = {8304, 8320, 185, 8321, 178, 8322, 179, 8323, 8308, 8324, 8309, 8325, 8310, 8326, 8311, 8327, 8312, 8328, 8313, 8329, 7491, 8336, 7495, 7580, 7496, 7497, 8337, 7584, 7501, 688, ...

6

Adding EOF When Creating The File If we are creating the CSV file ourselves then we can open a stream, export the CSV, and write the EOF character separately at the end: $data = {{1, 2, 3}, {4, 5, 6}};$stream = OpenWrite["test.csv"]; Export[$stream,$data, "CSV"]; WriteString[$stream, FromCharacterCode[26]]; Close[$stream]; We can verify that the result ...

6

The "Class" NetEncoder outputs one-hot vectors when the "UnitVector" form is specified: NetEncoder[{"Class", voc, "UnitVector"}][list] {1,0,0,0,0,0,0,0},{0,1,0,0,0,0,0,0},{0,0,1,0,0,0,0,0}, {0,0,0,1,0,0,0,0},{0,0,0,0,1,0,0,0},{0,0,0,0,0,1,0,0}, {1,0,0,0,0,0,0,0},{0,0,0,0,0,0,1,0},{0,0,0,0,0,0,0,1}}

5

I can't provide working code because I don't have access to Mathematica now, but you can do this using MakeExpression. You can reverse engineer the required box structure by typing the InfixForm of Times[x,y] in an Input Cell. Select the input cell, and press Shift-Ctrl-E to see the underlying box structure for infix multiplication. You want the same ...

5

I'm going to assume you want this specific char: Unicode Han Character The issue is not with Mathematica's character code conversion, but instead the encoding source. The value you have obtained is in Hex, where you need the decimal to use FromCharacterCode. Hex-> Dec-> Front end FromCharacterCode[Interpreter["HexInteger"]["5929"]] References 78666

Only top voted, non community-wiki answers of a minimum length are eligible