# Tag Info

22

Indeed, Hash has changed for 11.3 in several ways. The one relevant here is that strings are now hashed according to their UTF-8 representation. As a simple example of the difference, consider the following string, which has only one character. c = FromCharacterCode[217]; (* "\[CapitalUGrave]" *) Its UTF-8 representation consists of the two (decimal) ...

21

Declaration: This method for Windows is based on the .NET code from Todd Gayley's this wonderful answer. My .NET knowledge is absolutely ZERO, all credit goes to Todd. Code: The main idea is to extract the "Input"-style code string, convert it to the UTF-16 little endian form, which is the standard byte order in Windows, feed the bytes to system ...

20

I'm Riccardo, current developer of URLRead in WL and I have some experience working with encoding in WL. I would like to inform you that this is not a bug. In modern versions of mathematica we have ByteArray, and this is a representation of bytes. But for decades strings have been both bytes and "unicode" at the same time. The problem here is that all ...

19

Tracing the evaluation (Mathematica 11.1.1) shows that the string is passed to DeveloperReadRawJSONStream which actually produces the messages: DeveloperReadRawJSONStream[StringToStream@string, "IssueMessagesAs" -> Import] General::jsonoutofrangeunicode: Out of range unicode code point encountered. Import::jsoninvalidtoken: Invalid token ...

14

Here's how to add a permanent menu item and keyboard shortcut for Silvia's solution. (This can be easily adapted to Jens' solution for Mac OS.) 1. Add Package Copy Silvia's first code block that starts with Needs["NETLink"] to a file and save it as UniCodeCopy.m in one of the directories included in $Path. 2. Initialize the Front End Save the following ... 14 I tried random inputs until I got something to work. It looks like it has to do with character codes: SystemPrivateLookupNameByCode[200] "CapitalEGrave" And then we can plug in the reverse: SystemPrivateLookupCodeByName["CapitalEGrave"] 200 Edit by yode as J.M.'s comment Grid[DeleteCases[{#, FromCharacterCode[#], SystemPrivate... 12 I use a small web application for when there are too many to convert by hand. 12 For file operations two workarounds currently work: 1) Use slash / instead of double backslash in the paths to files under Windows: Import["D:/test/кириллическое название/файл.txt"] 2) Use FileNameJoin or StringJoin: Import[FileNameJoin[{"D:", "test", "кириллическое название", "файл.txt"}]] Import[StringJoin[{"D:", "\\", "test", "\\", "кириллическое ... 11 I have had some success with the following function that I wrote a couple of years ago to battle quoted printables on MathGroup: translateQuotedPrintable[str_String] := StringReplace[str, {"=" ~~ c1:HexadecimalCharacter~~c2:HexadecimalCharacter :> FromCharacterCode[FromDigits[c1 <> c2, 16], "Math1"],"=" ~~ EndOfLine -> ""}] ... 11 Here is a function that copies a Unicode string to the clipboard using JLink: Needs["JLink"]; InstallJava[]; LoadJavaClass["java.awt.Toolkit", AllowShortContext -> False]; uniclip[s_String] := JavaBlock[ javaawtToolkitgetDefaultToolkit[]@getSystemClipboard[]@setContents[#, #]& @ JavaNew["java.awt.datatransfer.StringSelection", s] ];... 11 Here's a version that doesn't require a temporary file. Linux (needs xclip) SetAttributes[copyUnicode, HoldAll]; copyUnicode[expr_] := With[{ stream = OpenWrite["!xclip -in -selection clipboard", CharacterEncoding -> "UTF-8"] }, WriteString[stream, ToString[Unevaluated@expr, InputForm]]; Close@stream; ]; Example: executing the cell Cell[... 11 The String format is binary, so there is no character encoding applied. If you use ISO8859-1 on https://www.base64decode.org, the output will match what you see in Mathematica. If you wish to Import as text with a character encoding, it can be done so like this: s="hqKtggfNEN7UOZG7rUlnIOsqDW+hstKUZ+3iOlltlDgE9GGZpLlvGAf/IJVUxAcNF3UBPvQa+GjGMeRdEFssng=="; ... 10 In my opinion you don't have many options here and honestly, I would not try to achieve this with Mathematica and Linux because the font-rendering was, is and will at least for some time be crappy. In the examples, I use the "Liberation Serif" which is the default serif font on my system. Let me give some ideas: The easiest thing I know is to use ... 10 That's a good question, though perhaps difficult to answer. The input aliases for these letters are stored not in the InputAliases option of the$FrontEnd object but rather in the UnicodeCharacters.tr file. Surely they are loaded into the Front End but I do not know the location of that data, and as such I cannot think of a clean way to access that mapping....

10

The different HTML entities are stored in SystemConvertMLStringDataDump$HTMLEntities on version 9 and from here, it's a simple StringReplace: StringReplace["<select></select>", SystemConvertMLStringDataDump`$HTMLEntities] (* "&lt;select&gt;&lt;/select&gt;" *)

10

That's because "Base64" is not a format but an encoding and you still need to tell Mathematica what format to import after decoding from Base64. This is described in the documentation. Try this: ImportString[s, {"Base64", "String"}]

9

Just in case somebody else needs it, here is a compiled answer. Thanks go out to 0x4A4D (for the actual solution), Michael Pilat (for the JLink part) and everybody else in here for the swift responses. Since this is apparently a bug of sorts in Mathematica 8, percent encoding the Greek letters in the URL will have to do. Reciting Michael Pilat's code ...

9

You have to specify the correct character encoding: Export["hello.txt", {t, r, θ, ϕ}, "List"]; old = Import["hello.txt", "List", CharacterEncoding -> "UTF8"] (* ==> {"t", "r", "θ", "ϕ"} *)

9

Following the comments above, I think I've managed to find the answer, that is, as m_goldberg and librik said, ReadList doesn't support character encoding, and maybe that's one of the reasons it's fast. However, that doesn't mean we can't make use of ReadList. In fact, following the advice from mfvonh, I found that Import internally uses ReadList to read ...

9

ExportAsASCII Currently I know only one well-documented and working (with exception for a few characters from Unicode Private Use Area which normally shouldn't appear in code) method to convert all special and non-ASCII characters in an arbitrary string into "PrintableASCII" programmatically: toInputForm[s_String] := StringTake[ ToString[s, InputForm, ...

9

Use Get["~/test.m", CharacterEncoding -> "UTF-8"] or change the value of $CharacterEncoding. Edit: Why the error message? What you are seeing is not a bug. Both UTF-8 and CP936 are variable-length encodings. Therefore, the way characters are (mis)interpreted depends on their exact sequence. With your original comment, your character and the closing ... 9 You cannot use the notation 16^^ with variable arguments, since it does not get "evaluated" like normal functions, but rather it is just a way of writing integers in different bases. For more information you can look at this question. A way to implement what you want would be hexToBinary[hexstring_String] := IntegerDigits[FromDigits[hexstring,16],2]... 8 It seems that the problem can be solved by setting explicit value of the CharacterEncoding global FE option (checked with MMa 8.0.4 and 9.0.0): SetOptions[$FrontEnd, CharacterEncoding -> "UTF8"]; Export["test.pdf", "кириллический текст"] An equivalent way (without changing the global FE settings): Export["test.pdf", Style["кириллический текст", ...

8

In the Option Inspector (CtrlShift O) look for AutoQuoteCharacters and set this on empty list. I presume you could set the same in the stylefile too.

8

Something like: ExportString[Cell[TextData["<select></select>"],"Text"],"HTML","FullDocument" -> False] produces: <p class="Text"> &lt;select&gt;&lt;/select&gt; </p> which might also be a good start.

8

If you are willing to use a simple string representation, then StringReplace might be adequate for your purpose: decode[s_] := StringReplace[s, {"1" -> "a", "01" -> "b", "00" -> "c"}] decode["110100"] (* "aabc" *) decode["101100101"] (* "abacab" *)

7

So PDF Import accepts a CharacterEncoding option, but it seems to ignore it. Exporting in UTF-8 or in ASCII gives the best results, with the former supporting the Euro symbol € and the latter supporting the ² and ³ symbols. Life is all about choice I guess. The String itself has to be encoded using Style, and the FontFamily matters, too: ImportString[ ...

7

m = {"computer", "экзамен", "elephant", "стол", "bread", "телефон", "exception", "desktop", "best", "колонка", "zoom", "saphire", "ярость"}; First Sort it: sortedm = SortBy[m, First@ToCharacterCode@# &] {"best", "bread", "computer", "desktop", "elephant", "exception", "saphire", "zoom", "колонка", "стол", "телефон", "экзамен", "ярость"} Then ...

7

Windows-specific: use short paths (short filenames, 8.3 file names) This method should work with all parts of the Mathematica system. Limitations short path can be found only for existing file/directory short paths exist only if generation of them isn't disabled by administrator (by default it is enabled) may not work for network paths this method isn't ...

7

You can use RemoveDiacritics: RemoveDiacritics /@ {"a", "\:1d62"} {"a", "i"}

Only top voted, non community-wiki answers of a minimum length are eligible