How to "Copy as Unicode" from a Notebook?

Question

I spent some time manually editing a post replacing Mathematica ASCII \[Alpha] with Unicode α. I did this by laboriously choosing Copy as LaTeX, pasting into the edit box, and then copying the Unicode symbol from the preview below. This made me realize I am lacking a "Copy as Unicode string" function in Mathematica.

How can I most easily copy an expression such as:

In Unicode:

αβ + Mod[δΨ, 2 ⁢ρ^2]

Answer 1

Since a native method is not forthcoming, I shall post my file based circumvention, for Windows.

You will need to have this utility in the command path (it apparently is stock with Windows 7).

copyUnicode[expr_] := Run["clip <",
   Export["$Clipboard.temp", ToString[expr, InputForm],
          "Text", CharacterEncoding -> "Unicode"] ];

Usage:

expr = \[Alpha]\[Beta] + Mod[\[Delta]\[CapitalPsi], 2\[InvisibleTimes]\[Rho]^2];

copyUnicode[expr]

This leaves the following text in the Windows Clipboard:

αβ + Mod[δΨ, 2*ρ^2]

---

Here is a version of the function that holds (does not evaluate) the expression:

SetAttributes[copyUnicode, HoldFirst]

copyUnicode[expr_, form_: InputForm] := 
  Run["clip <", 
   Export["$Clipboard.temp", ToString[Unevaluated@expr, form], "Text", 
    CharacterEncoding -> "Unicode"]];

Now:

Plot[\[Alpha], {\[Alpha], 0, 10}] // copyUnicode

Puts in the Windows Clipboard:

Plot[α, {α, 0, 10}]

Answer 2

Declaration:

This method for Windows is based on the .NET code from Todd Gayley's this wonderful answer. My .NET knowledge is absolutely ZERO, all credit goes to Todd.

Code:

The main idea is to extract the "Input"-style code string, convert it to the UTF-16 little endian form, which is the standard byte order in Windows, feed the bytes to system clipboard by using .NET API.

The main functions are placed in a package:

Needs["NETLink`"]
InstallNET[];
BeginPackage["UniCodeCopy`"]

mmaUnicodeToUTF16LE::usage = 
        ToString[Row[{Style["mmaUnicodeToUTF16LE[_Integer]", Bold], " convert integer list obtained from ", Style[Row[{"ToCharacterCode[", Style["string", Italic], ", \"Unicode\"]"}], Bold], " to integer list consistent with the UTF‐16 Little Endian standard."}], StandardForm];
stringToUTF16LE::usage = 
        ToString[Row[{Style["stringToUTF16LE[_String]", Bold], " convert ", Style["string", Italic], " to integer list consistent with the UTF‐16 Little Endian standard."}], StandardForm];
WriteToClipboardUnicode::usage = 
        ToString[Row[{Style["WriteToClipboardUnicode[_String]", Bold], " write ", Style["string", Italic], " to Windows clipboard using the UTF‐16 Little Endian encoding."}], StandardForm];

Begin["`Private`"]

Clear[mmaUnicodeToUTF16LE]
mmaUnicodeToUTF16LE[bytecode_Integer] := 
                    PadRight[#, 2 Ceiling[Length@#/2]] &@
                     Join[Most[#] - Rest[# 2^8], #[[{-1}]]] &@
        DeleteCases[FixedPointList[BitShiftRight[#, 8] &, bytecode], 0]

Clear[stringToUTF16LE]
stringToUTF16LE[str_String] := 
                    mmaUnicodeToUTF16LE /@ ToCharacterCode[str, "Unicode"] // 
            Flatten // Join[#, {0, 0}] &

Clear[WriteToClipboardUnicode]
WriteToClipboardUnicode[str_String] :=
       Module[{bytecode, strm, dataObject},
          bytecode = stringToUTF16LE[str];
          NETLink`NETBlock[
                           strm = NETLink`NETNew["System.IO.MemoryStream", bytecode];
                           dataObject = NETLink`NETNew["System.Windows.Forms.DataObject"];
                           dataObject@NETLink`SetData["Text", strm];
                           NETLink`LoadNETType["System.Windows.Forms.Clipboard"];
                           System`Windows`Forms`Clipboard`SetDataObject[dataObject]
                          ]
              ]

End[]
EndPackage[]

Then execute the following code, which will generate a palette with a UniCode Copy button:

Button["UniCode Copy",
    Module[{codestr},
        timestamp = {};
        FrontEndExecute[FrontEndToken["CopySpecial", "InputText"]];
        codestr = 
            NotebookGet[ClipboardNotebook[]][[1, 1, 1]] // 
                StringReplace[#, {"\\\n" -> "", 
                            "\n" ~~ space : " " ... :> 
                                StringJoin["\n", ConstantArray[space, 4]]}] &;
        UniCodeCopy`WriteToClipboardUnicode[codestr]
        ],
    Method -> "Queued"
    ]

SelectionMove[EvaluationNotebook[], Previous, Cell];
FrontEndExecute[FrontEndToken["GeneratePalette"]];
FrontEndExecute[FrontEndToken["Clear"]];

To use it, select the Cells you want to copy as input text code, then press the button. The package can be put in a file and be auto-loaded when MMA start, the palette can be installed to system menu.

Examples:

Unicode expressions in Mathematica Notebook:

Text generated directly by the above code:

"αβγδϵζηθχϕϡΔΥϖϒϠíãéõÂÆŁØÝ∟△♆♇√÷∇■○▫✶✓¥¶†︵│︷中文汉字日文コンピュータ"

α β + Mod[δΨ, 2 ρ^2]

Answer 3

Edit 2: A new version of the Mac solution with button is listed below

Fixed problem with pasting into textarea

In some applications on Mac, copying as Unicode from Mathematica already works without having to do any postprocessing. However, it doesn't work in textarea fields in web browsers.

Nevertheless, if you're willing to do a few additional mouse clicks, the Unicode forms can be brought into the browser by taking the detour through one of those applications that do support Mathematica's characters. One convenient choice is to open a new window in TextEdit and paste your original Mathematica code into it first. In the default RichText mode, TextEdit displays the special characters as Unicode glyphs. Then you can simply copy whatever you just pasted back from TextEdit and paste it into the browser. It seems that as soon as TextEdit is recognized as the provider of the Unicode text on the pasteboard, there is no conversion back to the Mathematica representation, so you get the correct appearance in the browser.

In the following, I use the same idea in order to automate the conversion. Instead of TextEdit, I go through a temporary file in RichText (RTF) format on the provider side. The Mac pasteboard does support RTF as a format for data on the pasteboard, but this isn't accepted by the browsers I've recently tried when attempting to paste into textarea.

Therefore, I have to add one additional step: using the textutil tool which is built into OS X, I convert the RTF file to a regular text file with Unicode characters first. Then I read the text file back in and put it on the pasteboard.

Of course, this means it's only going to work on Mac OS X because it uses Cocoa bindings in the built-in Python interpreter:

copyAsUnicode[t_] := 
 Module[{out = 
    FileNameJoin[{$TemporaryDirectory, 
      "MathematicaOutput" <> StringJoin[Map[ToString, DateList[]]]}]},
   Export[out <> ".rtf", t];
  Run["textutil -convert txt " <> out <> ".rtf -output " <> out <> 
    ".txt"]; 
  Run["printf \"from AppKit import *\n\
board=NSPasteboard.generalPasteboard()\n\
content=NSData.dataWithContentsOfFile_('" <> out <> 
    ".txt')\nboard.declareTypes_owner_([NSStringPboardType], None)\n\
board.setData_forType_(content, NSStringPboardType)\n\
\" | /usr/bin/python"];
  DeleteFile[{out <> ".txt", out <> ".rtf"}]]

The idea is to export to RTF and read the result to the clipboard outside of Mathematica. The function is invoked for example as copyAsUnicode["αβ+Mod[δΨ+ρ2]"]. This example itself was copied that way, too, i.e., I typed copyAsUnicode["copyAsUnicode[\"αβ+Mod[δΨ+ρ2]\"]"], which I again copied the same way... OK, I think you get the idea.

Of course the next step would be to make this into a Palette that acts on the NotebookSelection, but the above is the main step. Maybe someone else knows how to do something like this in other operating systems (I don't).

Installing this function as a button:

Responding to the comment, here I'm just listing the same function as above, but wrapped in Silvia's code to make it into a button. It requires no package loading because I inlined everything into the button code:

Button["UniCode Copy", Module[{codestr},
FrontEndExecute[FrontEndToken["CopySpecial", "InputText"]];
codestr = NotebookGet[ClipboardNotebook[]][[1, 1, 1]];
 Module[{out = 
    FileNameJoin[{$TemporaryDirectory, 
      "MathematicaOutput" <> StringJoin[Map[ToString, DateList[]]]}]},
   Export[out <> ".rtf", codestr];
  Run["textutil -convert txt " <> out <> ".rtf -output " <> out <> 
    ".txt"]; 
  Run["printf \"from AppKit import *\n\
board=NSPasteboard.generalPasteboard()\n\
content=NSData.dataWithContentsOfFile_('" <> out <> 
    ".txt')\nboard.declareTypes_owner_([NSStringPboardType], None)\n\
board.setData_forType_(content, NSStringPboardType)\n\
\" | /usr/bin/python"];
  DeleteFile[{out <> ".txt", out <> ".rtf"}]]],
Method -> "Queued"]
SelectionMove[EvaluationNotebook[], Previous, Cell];
FrontEndExecute[FrontEndToken["GeneratePalette"]];
FrontEndExecute[FrontEndToken["Clear"]];

Now you can also install this permanently by going to Palettes > Install Palette and selecting a name for the palette.

Answer 4

Here's how to add a permanent menu item and keyboard shortcut for Silvia's solution. (This can be easily adapted to Jens' solution for Mac OS.)

1. Add Package

Copy Silvia's first code block that starts with Needs["NETLink`"] to a file and save it as UniCodeCopy.m in one of the directories included in $Path.

2. Initialize the Front End

Save the following to the file $UserBaseDirectory/Autoload/FrontEnd/init.m (create it if needed):

Needs["UniCodeCopy`"]

CopySelectionAsUnicode[] :=
    Module[{selection, code},
           FrontEndExecute[FrontEndToken["CopySpecial", "InputText"]];
           selection = NotebookGet[ClipboardNotebook[]][[1, 1, 1]];
           code = StringReplace[selection, {"\\\n" -> "",
                                            "\n" ~~ space : " " ... :> StringJoin["\n", ConstantArray[space, 4]]}];
           UniCodeCopy`WriteToClipboardUnicode[code];]

FrontEndExecute[
    FrontEnd`AddMenuCommands["Copy",
                             {MenuItem["Copy as Unicode",
                                       FrontEndExecute[CopySelectionAsUnicode[]],
                                       (* Alt + U *)
                                       MenuKey["u", Modifiers -> {"Command"}],
                                       System`MenuEvaluator -> Automatic,
                                       Method -> "Queued"]}]]

For key bindings, the possible modifiers are "Shift", "Control", "Command" (Alt) and "Option" (Alt).

(It took me a few hours to figure this out. I hope this proves useful for other struggling novices.)

Answer 5

Here is a function that copies a Unicode string to the clipboard using JLink:

Needs["JLink`"];
InstallJava[];
LoadJavaClass["java.awt.Toolkit", AllowShortContext -> False];

uniclip[s_String] :=
  JavaBlock[
    java`awt`Toolkit`getDefaultToolkit[]@getSystemClipboard[]@setContents[#, #]& @
      JavaNew["java.awt.datatransfer.StringSelection", s]
  ];

From Mathematica 12.3 onward, we can use ExternalFunction instead of JLink (thanks to Ben Izd for pointing this out):

uniclip = ExternalFunction["Java",
  "static void uniclip(String string) {
     var s = new java.awt.datatransfer.StringSelection(string);
     java.awt.Toolkit.getDefaultToolkit().getSystemClipboard().setContents(s, s);
   }"
];

Example usage:

expr = \[Alpha]\[Beta] + Mod[\[Delta]\[CapitalPsi], 2\[InvisibleTimes]\[Rho]^2];

uniclip @ ToString[expr, InputForm]

After evaluating these expressions, the clipboard contains αβ + Mod[δΨ, 2*ρ^2].

Answer 6

I use a small web application for when there are too many to convert by hand.

Answer 7

Here's a version that doesn't require a temporary file.

Linux (needs xclip)

SetAttributes[copyUnicode, HoldAll];
copyUnicode[expr_] := With[{
      stream = OpenWrite["!xclip -in -selection clipboard", CharacterEncoding -> "UTF-8"]
   },
   WriteString[stream, ToString[Unevaluated@expr, InputForm]];
   Close@stream;
];

Example: executing the cell

Cell[BoxData[
 RowBox[{"copyUnicode", "[", 
  RowBox[{
   RowBox[{
    RowBox[{
     SuperscriptBox["x", "2"], "\[SmallCircle]", 
     RowBox[{"{", 
      RowBox[{
       RowBox[{"\[LeftFloor]", "\[Alpha]", "\[RightFloor]"}], ",", 
       "\"\<\[LeftFloor]\[Alpha]\[RightFloor]\>\""}], "}"}]}], 
    "\[PlusMinus]", 
    RowBox[{
     SqrtBox["5"], "\[CirclePlus]", "\[HappySmiley]"}]}], 
   "<=", 
   "\"\<\[Integral]\[PartialD]\[RightArrow]\[Union]\[Sum]\[Infinity]\
\[Element]\>\""}], "]"}]], "Input"]

gives x^2 ∘ {Floor[α], "⌊α⌋"} ± Sqrt[5] ⊕ ☺ <= "∫∂→⋃∑∞∈".

Windows

Not tested, but it should work if you use "!clip"instead of"!xclip -in -selection clipboard"`. You might have to change the encoding to UTF-16.

Caveats

• Note that certain characters get ASCIIfied anyway in InputForm when not inside a string.
• Mathematica uses non-standard private-use code points for some characters like U+211D ℝ, even when a standard code point exists, so the output will be wrong if the input contains such characters.

Answer 8

Here's a method that's really no different from the standard ones except it's generalized for copying cells and, theoretically, for Windows and Unix.

We'll use RunProcess because this exists now, and tweak the ProcessEnvironment to make it work for Mac:

FrontEnd`unicodeCopy[s_String,
   enc : _String | Automatic : Automatic,
   lang : _String | Automatic : Automatic
   ] :=
  With[{
    encoding = Replace[enc, Automatic :> $CharacterEncoding],
    language = Replace[lang, Automatic :> $Language]
    },
   If[#["ExitCode"] =!= 0,
      FrontEnd`unicodeCopy::copyerr = "error in copy process:\n ``";
      Message[FrontEnd`unicodeCopy::copyerr, #["StandardError"]];,
      #["StandardOuput"];
      ] &@
    RunProcess[
     Switch[$OperatingSystem,
      "Windows",
      "clip",
      "MacOSX",
      "pbcopy",
      "Unix",
      {"xsel", "-b"}
      ],
     All,
     ExportString[s,
      "String",
      CharacterEncoding -> encoding
      ],
     ProcessEnvironment -> <|
       "LANG" ->
        Replace[ToLowerCase@language,
          "english" -> "en_US"
          ] <> "." <>
         Replace[encoding, {
           "UTF8" -> "UTF-8",
           "Unicode" -> "UTF-16"
           }],
       "PATH" -> Environment["PATH"]
       |>
     ]
   ];

(I'm using FrontEnd` for when I dump to "MenuSetup.tr", just so it looks consistent)

Now we'll define a bunch of junk to generalize to make it work across an NB:

$unicodeTR =
  FrontEnd`FindFileOnPath["UnicodeCharacters.tr", 
    "PrivatePathsTextResources"] // FrontEndExecute;
$unicodeReplacements = #[[2]] -> 
     ToExpression[
      "\"" <> StringReplace[#[[1]], {"x" -> ":", 
         StartOfString ~~ "0" -> "\\"}] <> "\""] & /@
   Rest[
    StringSplit@StringSplit[Import[$unicodeTR, "String"], "\n"]
    ];

FrontEnd`UnicodeCopy[c : {__Cell}] :=
  Replace[
   MathLink`CallFrontEnd[
    ExportPacket[Cell[CellGroupData[c]], "InputText"]
    ], {
    {s_String, __} :>
     FrontEnd`unicodeCopy@
      StringReplace[s,$unicodeReplacements]
    }];
FrontEnd`UnicodeCopy[c_Cell] :=
  Replace[
   MathLink`CallFrontEnd[
    ExportPacket[c, "InputText"]
    ], {
    {s_String, __} :>
     FrontEnd`unicodeCopy@
      StringReplace[s,$unicodeReplacements]
    }];
FrontEnd`UnicodeCopy[b_BoxData | _TextData] :=

  FrontEnd`UnicodeCopy@Cell[b];
FrontEnd`UnicodeCopy[b_RowBox] :=

  FrontEnd`UnicodeCopy@Cell[BoxData@b];
FrontEnd`UnicodeCopy[
   nb : _NotebookObject | _FrontEnd`NotebookObject | \
_FrontEnd`SelectedNotebook | _FrontEnd`InputNotebook | \
_FrontEnd`ButtonNotebook | _FrontEnd`EvaluationNotebook | \
_FrontEnd`MessagesNotebook | _FrontEnd`HelpBrowserNotebook | \
_FrontEnd`ClipboardNotebook
   ] :=
  FrontEnd`UnicodeCopy[Evaluate@NotebookRead@nb];
FrontEnd`UnicodeCopy[
   nb : _InputNotebook | _EvaluationNotebook | _ClipboardNotebook | \
_ButtonNotebook | _MessagesNotebook | _HelpBrowserNotebook] :=

  FrontEnd`UnicodeCopy@Evaluate@nb;
FrontEnd`UnicodeCopy[Optional[Automatic, Automatic]] :=

  FrontEnd`UnicodeCopy@InputNotebook[];
FrontEnd`UnicodeCopy[s_String] :=
  FrontEnd`unicodeCopy[s];
FrontEnd`UnicodeCopy[e_] :=

  FrontEnd`UnicodeCopy[Evaluate@ToString[Unevaluated[e], InputForm]];
FrontEnd`UnicodeCopy~SetAttributes~HoldFirst

Then I'll add this to my MenuSetup using the framework I wrote up here and whose up-to-date implementation lives here

FEMenuSetupAdd[
 {"Edit", "Copy As", 5},
 "Unicode" -> KernelExecute[ToExpression["FrontEnd`UnicodeCopy[]"]],
 MenuEvaluator -> Automatic,
 System`MenuKey["C", System`Modifiers -> {"Control"}]
 ]

This now lives under the "Copy As" menu:

And it can be used by function:

FrontEnd`UnicodeCopy["asdasd\[Alpha]\[Beta]\[Gamma]"]

asdasdαβγ

FrontEnd`UnicodeCopy[Cell["asdasd\[Alpha]\[Beta]\[Gamma]"]]

asdasdαβγ

Or by Control-C:

FrontEnd`UnicodeCopy[Cell["asdasdαβγ"]]

(fixed, now, by doing a more selective replacement from UnicodeCharacters.tr)

One issue here is that I was a bit over zealous with some replacements when copying from cells and boxes (has to do with what comes out of the ExportPacket):

(Command-C): "asdasd\\[Alpha]\\[Beta]\\[Gamma]" 
(Control-C): "asdasd\α\β\γ" 

And just to include the original example:

\[Alpha]\[Beta] + 
 Mod[\[Delta]\[CapitalPsi], 2\[InvisibleTimes]\[Rho]^2]

becomes:

αβ + 
 Mod[δΨ, 2⁢ρ^2]

(note how, unfortunately, the added spacing from the cell copy is preserved)

Answer 9

Edit: Updated code

As a generalization to the excellent answer by ens, Silvia's solution also can be added as a palette to the menu as follows. First, create and save the UniCodeCopy.m package, as described by ens. Then create as a separate notebook, perhaps named Unicode Copy Source.nb,

NotebookPut[
    Notebook[{Cell[BoxData[ButtonBox["\"UniCode Copy\"", Appearance -> Automatic,
        ButtonFunction :> Module[{codestr},                  
        AppendTo[$Path, FileNameJoin[{$UserDocumentsDirectory, 
            "Mathematica/MyPackages"}]]; 
        Needs["UniCodeCopy`"];                  
        FrontEndExecute[FrontEndToken["CopySpecial", "InputText"]]; 
        codestr = (StringReplace[#1, {"\\\n" -> "", "\n" ~~ space : " " ... :> 
            StringJoin["\n", ConstantArray[space, 4]]}] & )
            [NotebookGet[ClipboardNotebook[]][[1, 1, 1]]]; 
            UniCodeCopy`WriteToClipboardUnicode[codestr]], 
            Evaluator -> Automatic, ImageSize -> 200, Method -> "Queued"]], 
            NotebookDefault]}, WindowSize -> {Fit, Fit}, 
            WindowMargins -> {{Automatic, 522}, {Automatic, 41}}, 
            WindowFrame -> "Palette", WindowElements -> {}, 
            StyleDefinitions -> "Palette.nb"], WindowTitle -> "UniCodeCopy"];
NotebookSave[%, FileNameJoin[{$UserDocumentsDirectory, 
    "Mathematica/MyPackages/UniCodeCopy.nb"}]]

Note that the third and fourth lines of code add the location of UniCodeCopy.m (in this case, my Mathematica/MyPackages directory) to $Path, if the location is not already there. (If it is there, these two lines of code can be omitted.) Executing Unicode Copy Source.nb creates the small palette, UnicodeCopy.nb, saved in Mathematica/MyPackages. Finally, use the Install Palette ... command in the Palette menu to register UnicodeCopy.nb as a palette. After Mathematica is restarted, it will display UnicodeCopy in the Palette menu.

Answer 10

Under windows with Nircmd available, one can add the code below (a modification of https://mathematica.meta.stackexchange.com/a/155/9754) to an initialization cell of
$UserBaseDirectory <> "\\Autoload\\FrontEnd\\init.m" to get Alt Gr+p as a shortcut for copying as unicode and having 4 spaces added at the beginning of each line.

FrontEndExecute[FrontEnd`AddMenuCommands["Copy", {MenuItem["Copy for MSE",
      FrontEndExecute[Module[{opts, content},
            If[MatchQ[NotebookRead[SelectedNotebook[]], {} | $Failed | NotebookRead[$Failed]], Beep[],

                opts = AbsoluteOptions[$FrontEndSession, "ExportTypesetOptions"];
                SetOptions[$FrontEndSession, "ExportTypesetOptions" -> {"PageWidth" -> 800, "CellBreaks" -> "\nCeLlBrEaK\n"}];
                FrontEndTokenExecute[SelectedNotebook[], "CopySpecial", "InputText"];
                content = NotebookGet[ClipboardNotebook[]];
                SetOptions[$FrontEndSession, opts];

                RunProcess[{"C:\\nircmd\\nircmdc.exe", "clipboard", "readfile", "\"" <> 
                          Export["C:\\nircmd\\mathCopy.txt", #, CharacterEncoding -> "Unicode"] <> "\""}] &[

                   Fold[StringReplace, StringJoin[{"    ", Cases[content, Cell[c_String, ___] :> c, All]}],
                      {{"" | "" -> "I", "" -> "E", "" | "" -> "<->", "\n" -> "\n    "},
                        {"    In[" ~~ NumberString ~~ "]:= " -> "    ",
                          "    CeLlBrEaK\n" | StartOfString ~~ "    Out[" ~~ NumberString ~~
                              "]= " ~~ e : Shortest[__] ~~ f : "\n    CeLlBrEaK" | EndOfString :>
                            "    (* " <> StringReplace[e, "\n    " -> "\n       "] <> " *)" <> StringTake[f, UpTo[5]],
                          "CeLlBrEaK" -> ""}}]]]]],
       MenuKey["p", Modifiers -> {"Control", "Command"}], System`MenuEvaluator -> Automatic, Method -> "Queued"]}]]