4
$\begingroup$

Background

Taking an example from Decoding GZIP encoded Body, BodyBytes (ByteArray) and BodyBytesArray from URLRead:

URLRead[
 "https://api.stackexchange.com/2.2/info?site=mathematica", "Headers"
]
{ ... 
, "content-type" -> "application/json; charset=utf-8"
, "content-encoding" -> "gzip"
, ...
}
Import @ "https://api.stackexchange.com/2.2/info?site=mathematica"

returns string json which can be later put to ImportString.

Problem

URLRead though throws a bunch of decoding errors suggesting it ignores "gzip" spec and goes directly to charset specified in content-type.

URLRead[
 "https://api.stackexchange.com/2.2/info?site=mathematica", "Body"
]

Workaround

is already shown in linked topic:

URLRead[
    "https://api.stackexchange.com/2.2/info?site=mathematica"
  , "BodyBytes"
] // FromCharacterCode // ImportString[#, {"gzip", "RawJSON"}] &

Question

Should that be the case? Is it a bug or am I missing the purpose of URLRead Body?

URLFetch behaves the same so I'm surprised it wasn't asked before.

URLFetch[
    "https://api.stackexchange.com/2.2/info?site=mathematica"
  , "Content"
]

Related

Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails

$\endgroup$
  • 2
    $\begingroup$ Note that Import actually does not need the "content-encoding" -> "gzip" header for recognizing gzip compressed data. You can check it with file=URLDownload["https://api.stackexchange.com/2.2/info?site=mathematica"];Import@file. The created file is a binary gzip-compressed file and Import recognizes the compression method from the first few bytes of the file, the HTTP "content-encoding" header isn't necessary at all. $\endgroup$ – Alexey Popkov Aug 29 '17 at 14:20
  • 1
    $\begingroup$ @AlexeyPopkov good point, so no one cares about content-encoding :) $\endgroup$ – Kuba Aug 29 '17 at 14:21
  • $\begingroup$ I would like to understand this too (+1). We knew that Import worked, from the quoted question.. BTW, "As of Version 11, URLFetch has been superseded by URLRead and URLExecute." $\endgroup$ – rhermans Aug 29 '17 at 14:23
  • 1
    $\begingroup$ @rhermans and Alexey, now it works, does it mean it was a bug? $\endgroup$ – Kuba Oct 17 '17 at 21:04
  • $\begingroup$ @Kuba I think it was a bug since ability to recognize gzip is of crucial importance for such function as URLRead. As I wrote above, I would expect it to recognize gzip even without the explicit "content-encoding" -> "gzip". Probably the latter can be checked using the file: protocol. $\endgroup$ – Alexey Popkov Oct 18 '17 at 9:52
2
$\begingroup$

As of V11.2 Content-Encoding specification is respected by URLRead and e.g.

URLRead[
  "https://api.stackexchange.com/2.2/info?site=mathematica", "Body"
]

Works well.

For earlier versions one needs to use a workaround like one shown above.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.