1
$\begingroup$

As we know the café is caf\xc3\xa9 after encode by UTF8:

>>> 'café'.encode('utf8')

b'caf\xc3\xa9'

But if I use

ToCharacterCode["café", "UTF8"]
ImportString["café", CharacterEncoding -> "UTF8"]

{99, 97, 102, 195, 169}

{99, 97, 102, 233}

This not what I after, how to get the real utf8 code(caf\xc3\xa9)?

$\endgroup$
  • 1
    $\begingroup$ 195 base 10 is c3 base 16. 169 base 10 is a9 base 16. The result seems correct to me? $\endgroup$ – Carl Lange May 7 at 18:24
  • 1
    $\begingroup$ if you want to print the ascii and use hex for others, something like If[# > 127, IntegerString[#, 16], FromCharacterCode[#]] & /@ ToCharacterCode["café", "UTF8"] $\endgroup$ – egwene sedai May 7 at 18:32
  • $\begingroup$ @CarlLange As your prompt, IntegerString[ToCharacterCode["café", "UTF8"], 16] can convert it. But mma have no easier method convert caf\xc3\xa9 into café mutually? $\endgroup$ – yode May 7 at 18:32
3
$\begingroup$

Your string b'caf\xc3\xa9' is the "real utf8 code" in some other language, not Mathematica. You can see the deconstructed UTF-8 in hexadecimal with:

BaseForm[ToCharacterCode["café", "UTF8"], 16]

If you have a bunch of such strings, understand that Mathematica is an expression rewriting language. This is a rewriting problem. A good way to do this is to convert your string to a more elementary expression and rewrite it.

hexReplace[l_List] := l //. {h___, 92, 120, a_, b_, t___} :> 
   {h, fromHexCode[a, b], t}

The replacement rule takes a list generated by ToCharacterCode, recognizes sequences starting with {92,120} ("\x"), feeds the following two codes to a function fromHexCode, and reassembles the list with the result inserted. ReplaceRepeated (//.) applies this transformation until it has found and replaced every match.

fromHexCode[a_, b_] := FromDigits[FromCharacterCode[{a, b}], 16]

fromHexCode takes a pair of integer character codes representing a pair of hex digits, and combines them into a single integer.

Now, compose a function that does the whole job:

hx2utf8[s_String] := FromCharacterCode[hexReplace[ToCharacterCode[s]], "UTF8"]

hx2utf8["caf\xc3\xa9"]
(* café *)
$\endgroup$
  • $\begingroup$ Thanks. since it is belong with other language, then caf\xc3\xa9 cannot convert into "café" as you said? $\endgroup$ – yode May 7 at 18:35
  • $\begingroup$ See edited answer. $\endgroup$ – John Doty May 7 at 20:13
  • $\begingroup$ Thanks a lot for your work. $\endgroup$ – yode May 8 at 5:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.