3
$\begingroup$

Bug persisting through 13.1.0 [CASE:4932406]


I'm using URLExecute to fetch some data from the arXiv preprint server but it seems that the encoding of the text I get is not correct.

Consider the following

text = URLExecute[
   "https://export.arxiv.org/api/query?search_query=%28cat:hep-th%29+\
AND+submittedDate:[20220310200000+TO+20220313200000]&start=0&max_\
results=500", "XML"];

The data returned by the server is an ATOM feed, so using XML as data format is justified. The encoding of the feed is UTF-8 according to

text[[1]]
(*{XMLObject[Declaration][Version->1.0,Encoding->UTF-8]}*)

However, some strings in this XML document are obviously wrongly encoded, e.g.

text[[2]][[3]][[25]][[3]][[6 ;; 8]]

which returns

{XMLElement["author", {}, {XMLElement["name", {}, {"Gábor Kasza"}]}], XMLElement["author", {}, {XMLElement["name", {}, {"LászlÃ\.b3 Csernai"}]}], XMLElement["author", {}, {XMLElement["name", {}, {"Tamás CsÃ\[Paragraph]rgÅ‘"}]}]}

Playing with the CharacterEncoding option of URLExecute doesn't seem to make any difference.

Unfortunately, I'm not an expert on text encoding and web programming so at this point it is not really clear to me if this is an issue of the arXiv server returning malformed data or a problem with Mathematica.

I'm using Mathematica 13.0 on Fedora 35 if that matters. Any help on getting the author names encoded correctly would be highly appreciated.

$\endgroup$
1
  • $\begingroup$ Looks like a bug to me. Have you reported it to the support? $\endgroup$ Apr 14 at 11:06

2 Answers 2

1
$\begingroup$

Looks like a bug to me. Please report it to the support.

Another workaround:

bytes = URLExecute[
   "https://export.arxiv.org/api/query?search_query=%28cat:hep-th%29+AND+submittedDate:[\
20220310200000+TO+20220313200000]&start=0&max_results=500", "Binary"];
xml = ImportString[FromCharacterCode[bytes, "UTF-8"], "XML"];
Cases[xml, "Gábor Kasza", Infinity]
{"Gábor Kasza"}

Alternatively:

xml = ImportString[ByteArrayToString[ByteArray[bytes]], "XML"];
Cases[xml, "Gábor Kasza", Infinity]
{"Gábor Kasza"}

Or using URLRead:

byteArray = 
  URLRead["https://export.arxiv.org/api/query?search_query=%28cat:hep-th%29+AND+\
submittedDate:[20220310200000+TO+20220313200000]&start=0&max_results=500", 
   "BodyByteArray"];

xml = ImportString[ByteArrayToString[byteArray], "XML"];

Cases[xml, "Gábor Kasza", Infinity]
{"Gábor Kasza"}

This method seems to be the most general workaround for such bugs in Mathematica: get the contents as bytes, not as string, then convert them to a string in the correct encoding, then import the latter.

$\endgroup$
1
  • 1
    $\begingroup$ Many thanks for multiple workarounds and the confirmation. I reported the bug to WRI [CASE:4932406] $\endgroup$
    – vsht
    Apr 14 at 13:36
1
$\begingroup$

For some reason, applying ToCharacterCode and then FromCharacterCode to the problematic strings seems to fix the encoding:

FromCharacterCode[ToCharacterCode["Gábor Kasza"], "UTF-8"]

returns

"Gábor Kasza"

Hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.