A bug in converting some Unicode characters into FullForm?

Question

Currently I know only one well-documented way to convert all special characters in an arbitrary string into their FullForm:

In InputForm, all special characters are written out fully when using "PrintableASCII".

ToString["Lamé \[LongRightArrow] \[Alpha]\[Beta]+", InputForm, 
   CharacterEncoding -> "PrintableASCII"]

"\"Lam\\[EAcute] \\[LongRightArrow] \\[Alpha]\\[Beta]+\""

But exhaustive checking reveals that for some characters this produces wrong results:

toInputForm[s_String] := 
  StringTake[ToString[s, InputForm, CharacterEncoding -> "PrintableASCII"], {2, -2}];
chs = {#, toInputForm[FromCharacterCode[#]]} & /@ Range[128, 65535];
sel = Select[chs, StringLength[#[[2]]] == 1 &];

Grid[Flatten[Transpose /@ Partition[sel, UpTo@11], 1], Frame -> {All, {{True, False}}}, 
 Spacings -> 0.2, FrameStyle -> LightGray]

The outputted characters correspond to the ASCII set, not to the original Unicode code points:

MinMax@ToCharacterCode@StringJoin@sel[[;; , 2]]

{0, 127}

Is there a fix for this? Or is there a robust and efficient way to convert all the special and Unicode characters into their corresponding FullForms?

---

P.S. An attempt to start from FullForm also produces wrong result:

ToString[FullForm@StringJoin@sel[[;; , 2]], OutputForm]

"\"\\000\\001\\002\\003\\004\\005\\006\\007\\b\\t\\013\\f\\016\\017\\020\\021\\022\\023\\\
024\\025\\026\\027\\030\\031\\032\\[RawEscape]\\034\\035\\036\\037\\177\""

Exporting as "Package" produces the correct output but it contains ugly header and several redundant newlines:

ExportString[FromCharacterCode[Range[61952, 61983]~Join~{62079}], "Package", 
 PageWidth -> Infinity]

"(* Created with the Wolfram Language : www.wolfram.com *)

\"\\:f200\\:f201\\:f202\\:f203\\:f204\\:f205\\:f206\\:f207\\:f208\\:f209\\:f20a\\:f20b\\:\
f20c\\:f20d\\:f20e\\:f20f\\:f210\\:f211\\:f212\\:f213\\:f214\\:f215\\:f216\\:f217\\:f218\\\
:f219\\:f21a\\:f21b\\:f21c\\:f21d\\:f21e\\:f21f\\:f27f\"

"

I'm hoping for something more elegant than cleaning out this...

Answer 1

I've contacted the support on this issue ([CASE:3933761]) but received nothing better than a suggestion to manually clean up the "Package" string using StringDrop. The following is my attempt to develop a workaround based on the "Package" export.

First, we can remove the package header by adding "Comments" -> None:

string = FromCharacterCode[Range[61952, 61983]];
p = ExportString[string, "Package", PageWidth -> Infinity, "Comments" -> None]

"\"\\:f200\\:f201\\:f202\\:f203\\:f204\\:f205\\:f206\\:f207\\:f208\\:f209\\:f20a\\:f20b\\:\
f20c\\:f20d\\:f20e\\:f20f\\:f210\\:f211\\:f212\\:f213\\:f214\\:f215\\:f216\\:f217\\:f218\\\
:f219\\:f21a\\:f21b\\:f21c\\:f21d\\:f21e\\:f21f\"\r\n"

Tracing ExportString and experimenting shows that basically it is equivalent to the following:

Module[{stream = Developer`StreamToString[CharacterEncoding -> "ASCII",
    FormatType -> InputForm, BinaryFormat -> True], result},
 Write[stream, string];
 result = Developer`StringFromStream@stream;
 Close[stream];
 result]

(here BinaryFormat -> True is added mostly due to this feature).

It is important that unlike ExportString this method produces under Windows a string ending with \"\n, not with \"\r\n what simplifies further clean-up.

The main problem with this method is that it leaves the ASCII characters untouched, so the control ASCII characters won't be converted into their printable representation.

But it does work correctly for all non-ASCII characters:

inputForm[string_String] := 
 Module[{stream = Developer`StreamToString[CharacterEncoding -> "ASCII", 
     FormatType -> InputForm, BinaryFormat -> True], result},
  Write[stream, string];
  result = StringDrop[Developer`StringFromStream@stream, -1];
  Close[stream];
  result]

check = With[{if = inputForm[#]}, ToExpression[if] === # && StringLength[if] > 3] & /@ 
  FromCharacterCode /@ Range[128, 65535];

And @@ check

True

(checked with Mathematica 10.4.1 and 11.1.1; this code crashes Kernel of version 8.0.4).

The CharacterEncoding -> "PrintableASCII" option value is supported by Developer`StreamToString but it contains exactly the same bug as the documented method shown in the question.

The above partial solution is equivalent to documented and more stable ToString method which should be preferred:

inputForm[s_String] := ToString[s, InputForm, CharacterEncoding -> "ASCII"];

The non-printable ASCII characters can be manually replaced with their FullForms using StringReplace.