UTF8 defines a way to map from bytes to glyphs. In particular, glyphs with code <128 can be 1 byte, glyphs with code <2048 can be 2 bytes, etc. up to 4 bytes. For instance, the white queen has code 9813 and is is encoded as the three bytes {226,153,149}.

I want to pass a String object containing such a byte array to FrontEnd`CopyToClipboard and/or other functions. However, for many 'named' characters, calling ByteArrayToString@ByteArray autoconverts the byte array to an escape sequence such as \[WhiteQueen] in this example.


How can I convert a ByteArray to a String with no internal processing or replacements? A hypothetical function barr2str should work such that

MathLink`CallFrontEnd@FrontEnd`CopyToClipboard@ barr2str@{226,153,149}

always yields a unicode character instead of an escape sequence. Do we have to resort to compiled c passing strategies (and if so, how)?


Following https://en.wikipedia.org/wiki/UTF-8#Encoding, here is a function that converts from a unicode codepoint to an appropriate byte array

unicode2barr[codepoint_] := If[# < 128, {#}, 
  FromDigits[PadLeft[Prepend[#, 0], 8, 1], 2] & /@ 
   Prepend[Partition[#2, 6], #] & @@ 
    TakeDrop[PadLeft[#, 1 + #2], 7 - #2/5] &[#, 
     Ceiling[Length@# - 1, 5]] &@IntegerDigits[#, 2]] &@codepoint

It may be tested as follows:

{FromCharacterCode@#, ByteArrayToString@ByteArray@unicode2barr@#}&\
  /@{946, FromDigits["20AC",16], 9813, 78064 (* your favorite codepoints *)}


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.