29

Since a native method is not forthcoming, I shall post my file based circumvention, for Windows. You will need to have this utility in the command path (it apparently is stock with Windows 7). copyUnicode[expr_] := Run["clip <", Export["$Clipboard.temp", ToString[expr, InputForm], "Text", CharacterEncoding -> "Unicode"] ]; Usage: expr =...


21

Declaration: This method for Windows is based on the .NET code from Todd Gayley's this wonderful answer. My .NET knowledge is absolutely ZERO, all credit goes to Todd. Code: The main idea is to extract the "Input"-style code string, convert it to the UTF-16 little endian form, which is the standard byte order in Windows, feed the bytes to system clipboard ...


21

Indeed, Hash has changed for 11.3 in several ways. The one relevant here is that strings are now hashed according to their UTF-8 representation. As a simple example of the difference, consider the following string, which has only one character. c = FromCharacterCode[217]; (* "\[CapitalUGrave]" *) Its UTF-8 representations consists of the two (decimal) ...


18

See if this helps: Needs["JLink`"]; ClearAll[toUpperCase]; toUpperCase[s_String] := JavaBlock[JavaNew["java.lang.String", s]@toUpperCase[]];


17

Tracing the evaluation (Mathematica 11.1.1) shows that the string is passed to Developer`ReadRawJSONStream which actually produces the messages: Developer`ReadRawJSONStream[StringToStream@string, "IssueMessagesAs" -> Import] General::jsonoutofrangeunicode: Out of range unicode code point encountered. Import::jsoninvalidtoken: Invalid token ...


16

I'm Riccardo, current developer of URLRead in WL and I have some experience working with encoding in WL. I would like to inform you that this is not a bug. In modern versions of mathematica we have ByteArray, and this is a representation of bytes. But for decades strings have been both bytes and "unicode" at the same time. The problem here is that all ...


15

Edit 2: A new version of the Mac solution with button is listed below Fixed problem with pasting into textarea In some applications on Mac, copying as Unicode from Mathematica already works without having to do any postprocessing. However, it doesn't work in textarea fields in web browsers. Nevertheless, if you're willing to do a few additional mouse ...


14

Here's how to add a permanent menu item and keyboard shortcut for Silvia's solution. (This can be easily adapted to Jens' solution for Mac OS.) 1. Add Package Copy Silvia's first code block that starts with Needs["NETLink`"] to a file and save it as UniCodeCopy.m in one of the directories included in $Path. 2. Initialize the Front End Save the following ...


14

I tried random inputs until I got something to work. It looks like it has to do with character codes: System`Private`LookupNameByCode[200] "CapitalEGrave" And then we can plug in the reverse: System`Private`LookupCodeByName["CapitalEGrave"] 200 Edit by yode as J.M.'s comment Grid[DeleteCases[{#, FromCharacterCode[#], System`Private`...


12

For file operations two workarounds currently work: 1) Use slash / instead of double backslash in the paths to files under Windows: Import["D:/test/кириллическое название/файл.txt"] 2) Use FileNameJoin or StringJoin: Import[FileNameJoin[{"D:", "test", "кириллическое название", "файл.txt"}]] Import[StringJoin[{"D:", "\\", "test", "\\", "кириллическое ...


11

Here is a function that copies a Unicode string to the clipboard using JLink: Needs["JLink`"]; InstallJava[]; LoadJavaClass["java.awt.Toolkit", AllowShortContext -> False]; uniclip[s_String] := JavaBlock[ java`awt`Toolkit`getDefaultToolkit[]@getSystemClipboard[]@setContents[#, #]& @ JavaNew["java.awt.datatransfer.StringSelection", s] ];...


11

I use a small web application for when there are too many to convert by hand.


11

Here's a version that doesn't require a temporary file. Linux (needs xclip) SetAttributes[copyUnicode, HoldAll]; copyUnicode[expr_] := With[{ stream = OpenWrite["!xclip -in -selection clipboard", CharacterEncoding -> "UTF-8"] }, WriteString[stream, ToString[Unevaluated@expr, InputForm]]; Close@stream; ]; Example: executing the cell Cell[...


11

I have had some success with the following function that I wrote a couple of years ago to battle quoted printables on MathGroup: translateQuotedPrintable[str_String] := StringReplace[str, {"=" ~~ c1:HexadecimalCharacter~~c2:HexadecimalCharacter :> FromCharacterCode[FromDigits[c1 <> c2, 16], "Math1"],"=" ~~ EndOfLine -> ""}] ...


10

That's a good question, though perhaps difficult to answer. The input aliases for these letters are stored not in the InputAliases option of the $FrontEnd object but rather in the UnicodeCharacters.tr file. Surely they are loaded into the Front End but I do not know the location of that data, and as such I cannot think of a clean way to access that mapping....


10

In my opinion you don't have many options here and honestly, I would not try to achieve this with Mathematica and Linux because the font-rendering was, is and will at least for some time be crappy. In the examples, I use the "Liberation Serif" which is the default serif font on my system. Let me give some ideas: The easiest thing I know is to use ...


10

The different HTML entities are stored in System`Convert`MLStringDataDump`$HTMLEntities on version 9 and from here, it's a simple StringReplace: StringReplace["<select></select>", System`Convert`MLStringDataDump`$HTMLEntities] (* "&lt;select&gt;&lt;/select&gt;" *)


10

That's because "Base64" is not a format but an encoding and you still need to tell Mathematica what format to import after decoding from Base64. This is described in the documentation. Try this: ImportString[s, {"Base64", "String"}]


10

The String format is binary, so there is no character encoding applied. If you use ISO8859-1 on https://www.base64decode.org, the output will match what you see in Mathematica. If you wish to Import as text with a character encoding, it can be done so like this: s="hqKtggfNEN7UOZG7rUlnIOsqDW+hstKUZ+3iOlltlDgE9GGZpLlvGAf/IJVUxAcNF3UBPvQa+GjGMeRdEFssng=="; ...


9

Reposting John Fultz’s comment above as a “community wiki” answer for everyone to improve: Mathematica simply has no support for non-plane-0 characters. That it appears to temporarily work should not fool you into thinking that M-- knows anything about such values. Those who saw the R&D keynote at the 2011 Tech Conference may remember my ...


9

Just in case somebody else needs it, here is a compiled answer. Thanks go out to 0x4A4D (for the actual solution), Michael Pilat (for the JLink part) and everybody else in here for the swift responses. Since this is apparently a bug of sorts in Mathematica 8, percent encoding the Greek letters in the URL will have to do. Reciting Michael Pilat's code ...


9

You have to specify the correct character encoding: Export["hello.txt", {t, r, θ, ϕ}, "List"]; old = Import["hello.txt", "List", CharacterEncoding -> "UTF8"] (* ==> {"t", "r", "θ", "ϕ"} *)


9

Following the comments above, I think I've managed to find the answer, that is, as m_goldberg and librik said, ReadList doesn't support character encoding, and maybe that's one of the reasons it's fast. However, that doesn't mean we can't make use of ReadList. In fact, following the advice from mfvonh, I found that Import internally uses ReadList to read ...


9

ExportAsASCII Currently I know only one well-documented and working (with exception for a few characters from Unicode Private Use Area which normally shouldn't appear in code) method to convert all special and non-ASCII characters in an arbitrary string into "PrintableASCII" programmatically: toInputForm[s_String] := StringTake[ ToString[s, InputForm, ...


9

Use Get["~/test.m", CharacterEncoding -> "UTF-8"] or change the value of $CharacterEncoding. Edit: Why the error message? What you are seeing is not a bug. Both UTF-8 and CP936 are variable-length encodings. Therefore, the way characters are (mis)interpreted depends on their exact sequence. With your original comment, your character and the closing ...


8

In the Option Inspector (CtrlShift O) look for AutoQuoteCharacters and set this on empty list. I presume you could set the same in the stylefile too.


8

It seems that the problem can be solved by setting explicit value of the CharacterEncoding global FE option (checked with MMa 8.0.4 and 9.0.0): SetOptions[$FrontEnd, CharacterEncoding -> "UTF8"]; Export["test.pdf", "кириллический текст"] An equivalent way (without changing the global FE settings): Export["test.pdf", Style["кириллический текст", ...


8

Something like: ExportString[Cell[TextData["<select></select>"],"Text"],"HTML","FullDocument" -> False] produces: <p class="Text"> &lt;select&gt;&lt;/select&gt; </p> which might also be a good start.


8

If you are willing to use a simple string representation, then StringReplace might be adequate for your purpose: decode[s_] := StringReplace[s, {"1" -> "a", "01" -> "b", "00" -> "c"}] decode["110100"] (* "aabc" *) decode["101100101"] (* "abacab" *)


Only top voted, non community-wiki answers of a minimum length are eligible