1
$\begingroup$

Problem context

I'm updating my working Mathematica Khan Academy reporting application. Since updating to 11.2 I'm having some errors importing via the Khan Academy API as JSON. Update: This particular issue is broken in 11.0.1 and 11.2

Specifically, when making the same query with different "ListId" values, each representing a class of students I'm "coaching" in Khan Academy, one class works and the other returns an error: "Cannot import Data as JSON format"

What I've tried

link = "https://www.khanacademy.org/api/internal/user/students/\
progress?dt_start=2017-10-22T07:00:00.000Z&dt_end=2017-10-29T06:59:59.\
999Z&listId=\
ag5zfmtoYW4tYWNhZGVteXJsCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6Ly9pZC\
5raGFuYWNhZGVteS5vcmcvYThlN2M3MWFiYjliNGJhZjg3OWM5ZTIxODQwMWQ5NWIMCxIL\
U3R1ZGVudExpc3QYgICAgKCOgQoM&studentsPerRequest=50&extended=1&tz_\
offset=-420&lang=en";

URLRead[HTTPRequest[
  OAuthSigning`Private`HMACSha1SignatureService[link, "HMAC", 
   "GET", $consumerKey, $consumerSecret, $accessTokenParams[
    "oauth_token"], $accessTokenParams["oauth_token_secret"]]]]

Returns:

enter image description here

Import[HTTPRequest[

  OAuthSigning`Private`HMACSha1SignatureService[link, "HMAC", 
   "GET", $consumerKey, $consumerSecret, $accessTokenParams[
    "oauth_token"], $accessTokenParams["oauth_token_secret"]]
          ]]

Returns:

enter image description here

URLExecute[HTTPRequest[

  OAuthSigning`Private`HMACSha1SignatureService[link, "HMAC", 
   "GET", $consumerKey, $consumerSecret, $accessTokenParams[
    "oauth_token"], $accessTokenParams["oauth_token_secret"]]
          ]]

Returns:

enter image description here

And when I click on the link, in the web it says, "OAuth error. Nonce already used: 15Y3zVd9cGGR6jQjGcogv043"

But!

When I try this with another link, which only contains a different ListId (representing a different class of students I'm managing), everything works as expected. I am able to use other API endpoints with this class, without error so I know that the ListId is valid. Also, its strange but true, Khan Academy encourages the use of undocumented, private API endpoints.

$\endgroup$
  • $\begingroup$ JSON can sometimes break when importing characters not in UTF-8. Try using Import[#, "JSON", "StrictDecoding" -> False]. $\endgroup$ – b3m2a1 Oct 29 '17 at 1:43
  • $\begingroup$ Thanks @b3m2a1 I just tried it and it didn't work. Any ideas why I'd get the nonce already used error? $\endgroup$ – Matt Green Oct 29 '17 at 1:47
  • $\begingroup$ That's the second time you're trying to read from the URL. Try generating the URL with the OAuthSigning`... call and simply going there. Also what do the guts of the HTTPResponse look like? (qualitatively, i.e. is there data cached in it that you can work look at to see what's gone wrong) $\endgroup$ – b3m2a1 Oct 29 '17 at 1:51
  • $\begingroup$ Okay @b3m2a1 I just tried this: gist.github.com/matthewalangreen/… and I'm still getting the cannot import data as JSON format error $\endgroup$ – Matt Green Oct 29 '17 at 1:55
  • $\begingroup$ Try importing as "Text" to see what could be messing up the JSON import $\endgroup$ – b3m2a1 Oct 29 '17 at 2:00
2
$\begingroup$

Thanks to @b3m2a1 for all the help, we were able to get this to work:

link = "https://www.khanacademy.org/api/internal/user/students/\
progress?dt_start=2017-10-22T07:00:00.000Z&dt_end=2017-10-29T06:59:59.\
999Z&listId=\
ag5zfmtoYW4tYWNhZGVteXJsCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6Ly9pZC\
5raGFuYWNhZGVteS5vcmcvYThlN2M3MWFiYjliNGJhZjg3OWM5ZTIxODQwMWQ5NWIMCxIL\
U3R1ZGVudExpc3QYgICAgKCOgQoM&studentsPerRequest=50&extended=1&tz_\
offset=-420&lang=en";


httpReq = 
  HTTPRequest[
   OAuthSigning`Private`HMACSha1SignatureService[link, "HMAC", 
    "GET", $consumerKey, $consumerSecret, $accessTokenParams[
     "oauth_token"], $accessTokenParams["oauth_token_secret"]]];

data = ImportString[ExportString[Import[httpReq, "Text"], "Text"], 
   "JSON"];

I'm still working to understand the details, but it seems that simply importing the data as JSON throws an error that ExportString handles for us and allows the data to import. I'd love to understand this further if y'all have any ideas.

$\endgroup$
  • 2
    $\begingroup$ This answer can be relevant. $\endgroup$ – Alexey Popkov Oct 30 '17 at 6:46
  • $\begingroup$ Can you tell us what headers you get? Especially content-type and content-encoding. $\endgroup$ – Kuba Oct 30 '17 at 7:32
  • $\begingroup$ This is what I get from HTTPResponse[HTTPReq] "application/json;charset=utf-8". @Kuba. Do you need more info? $\endgroup$ – Matt Green Oct 30 '17 at 18:53
  • $\begingroup$ @MattGreen I guess the issue it the same as in linked topic. Nothing more to add here. But you can check if this isn't a faster workaround: ImportString[ToCharacterCode @ URLRead[..., "BodyBytes"], "JSON"] $\endgroup$ – Kuba Nov 2 '17 at 7:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.