Character encoding of the output from URLExecute

Question

Bug persisting through 13.1.0 [CASE:4932406]

---

I'm using URLExecute to fetch some data from the arXiv preprint server but it seems that the encoding of the text I get is not correct.

Consider the following

text = URLExecute[
   "https://export.arxiv.org/api/query?search_query=%28cat:hep-th%29+\
AND+submittedDate:[20220310200000+TO+20220313200000]&start=0&max_\
results=500", "XML"];

The data returned by the server is an ATOM feed, so using XML as data format is justified. The encoding of the feed is UTF-8 according to

text[[1]]
(*{XMLObject[Declaration][Version->1.0,Encoding->UTF-8]}*)

However, some strings in this XML document are obviously wrongly encoded, e.g.

text[[2]][[3]][[25]][[3]][[6 ;; 8]]

which returns

{XMLElement["author", {}, {XMLElement["name", {}, {"Gábor Kasza"}]}], XMLElement["author", {}, {XMLElement["name", {}, {"LászlÃ\.b3 Csernai"}]}], XMLElement["author", {}, {XMLElement["name", {}, {"Tamás CsÃ\[Paragraph]rgÅ‘"}]}]}

Playing with the CharacterEncoding option of URLExecute doesn't seem to make any difference.

Unfortunately, I'm not an expert on text encoding and web programming so at this point it is not really clear to me if this is an issue of the arXiv server returning malformed data or a problem with Mathematica.

I'm using Mathematica 13.0 on Fedora 35 if that matters. Any help on getting the author names encoded correctly would be highly appreciated.

Answer 1

Looks like a bug to me. Please report it to the support.

Another workaround:

bytes = URLExecute[
   "https://export.arxiv.org/api/query?search_query=%28cat:hep-th%29+AND+submittedDate:[\
20220310200000+TO+20220313200000]&start=0&max_results=500", "Binary"];
xml = ImportString[FromCharacterCode[bytes, "UTF-8"], "XML"];
Cases[xml, "Gábor Kasza", Infinity]

{"Gábor Kasza"}

Alternatively:

xml = ImportString[ByteArrayToString[ByteArray[bytes]], "XML"];
Cases[xml, "Gábor Kasza", Infinity]

{"Gábor Kasza"}

Or using URLRead:

byteArray = 
  URLRead["https://export.arxiv.org/api/query?search_query=%28cat:hep-th%29+AND+\
submittedDate:[20220310200000+TO+20220313200000]&start=0&max_results=500", 
   "BodyByteArray"];

xml = ImportString[ByteArrayToString[byteArray], "XML"];

Cases[xml, "Gábor Kasza", Infinity]

{"Gábor Kasza"}

This method seems to be the most general workaround for such bugs in Mathematica: get the contents as bytes, not as string, then convert them to a string in the correct encoding, then import the latter.

Answer 2

For some reason, applying ToCharacterCode and then FromCharacterCode to the problematic strings seems to fix the encoding:

FromCharacterCode[ToCharacterCode["Gábor Kasza"], "UTF-8"]

returns

"Gábor Kasza"

Hope this helps.