1
$\begingroup$

Problem context

I'm updating my working Mathematica Khan Academy reporting application. Since updating to 11.2 I'm having some errors importing via the Khan Academy API as JSON. Update: This particular issue is broken in 11.0.1 and 11.2

Specifically, when making the same query with different "ListId" values, each representing a class of students I'm "coaching" in Khan Academy, one class works and the other returns an error: "Cannot import Data as JSON format"

What I've tried

link = "https://www.khanacademy.org/api/internal/user/students/\
progress?dt_start=2017-10-22T07:00:00.000Z&dt_end=2017-10-29T06:59:59.\
999Z&listId=\
ag5zfmtoYW4tYWNhZGVteXJsCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6Ly9pZC\
5raGFuYWNhZGVteS5vcmcvYThlN2M3MWFiYjliNGJhZjg3OWM5ZTIxODQwMWQ5NWIMCxIL\
U3R1ZGVudExpc3QYgICAgKCOgQoM&studentsPerRequest=50&extended=1&tz_\
offset=-420&lang=en";

URLRead[HTTPRequest[
  OAuthSigning`Private`HMACSha1SignatureService[link, "HMAC", 
   "GET", $consumerKey, $consumerSecret, $accessTokenParams[
    "oauth_token"], $accessTokenParams["oauth_token_secret"]]]]

Returns:

enter image description here

Import[HTTPRequest[

  OAuthSigning`Private`HMACSha1SignatureService[link, "HMAC", 
   "GET", $consumerKey, $consumerSecret, $accessTokenParams[
    "oauth_token"], $accessTokenParams["oauth_token_secret"]]
          ]]

Returns:

enter image description here

URLExecute[HTTPRequest[

  OAuthSigning`Private`HMACSha1SignatureService[link, "HMAC", 
   "GET", $consumerKey, $consumerSecret, $accessTokenParams[
    "oauth_token"], $accessTokenParams["oauth_token_secret"]]
          ]]

Returns:

enter image description here

And when I click on the link, in the web it says, "OAuth error. Nonce already used: 15Y3zVd9cGGR6jQjGcogv043"

But!

When I try this with another link, which only contains a different ListId (representing a different class of students I'm managing), everything works as expected. I am able to use other API endpoints with this class, without error so I know that the ListId is valid. Also, its strange but true, Khan Academy encourages the use of undocumented, private API endpoints.

Improve this question
$\endgroup$
7
  • $\begingroup$ JSON can sometimes break when importing characters not in UTF-8. Try using Import[#, "JSON", "StrictDecoding" -> False]. $\endgroup$
    – b3m2a1
    Oct 29, 2017 at 1:43
  • $\begingroup$ Thanks @b3m2a1 I just tried it and it didn't work. Any ideas why I'd get the nonce already used error? $\endgroup$
    – Matt Green
    Oct 29, 2017 at 1:47
  • $\begingroup$ That's the second time you're trying to read from the URL. Try generating the URL with the OAuthSigning`... call and simply going there. Also what do the guts of the HTTPResponse look like? (qualitatively, i.e. is there data cached in it that you can work look at to see what's gone wrong) $\endgroup$
    – b3m2a1
    Oct 29, 2017 at 1:51
  • $\begingroup$ Okay @b3m2a1 I just tried this: gist.github.com/matthewalangreen/… and I'm still getting the cannot import data as JSON format error $\endgroup$
    – Matt Green
    Oct 29, 2017 at 1:55
  • $\begingroup$ Try importing as "Text" to see what could be messing up the JSON import $\endgroup$
    – b3m2a1
    Oct 29, 2017 at 2:00

1 Answer 1

Reset to default
2
$\begingroup$

Thanks to @b3m2a1 for all the help, we were able to get this to work:

link = "https://www.khanacademy.org/api/internal/user/students/\
progress?dt_start=2017-10-22T07:00:00.000Z&dt_end=2017-10-29T06:59:59.\
999Z&listId=\
ag5zfmtoYW4tYWNhZGVteXJsCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6Ly9pZC\
5raGFuYWNhZGVteS5vcmcvYThlN2M3MWFiYjliNGJhZjg3OWM5ZTIxODQwMWQ5NWIMCxIL\
U3R1ZGVudExpc3QYgICAgKCOgQoM&studentsPerRequest=50&extended=1&tz_\
offset=-420&lang=en";


httpReq = 
  HTTPRequest[
   OAuthSigning`Private`HMACSha1SignatureService[link, "HMAC", 
    "GET", $consumerKey, $consumerSecret, $accessTokenParams[
     "oauth_token"], $accessTokenParams["oauth_token_secret"]]];

data = ImportString[ExportString[Import[httpReq, "Text"], "Text"], 
   "JSON"];

I'm still working to understand the details, but it seems that simply importing the data as JSON throws an error that ExportString handles for us and allows the data to import. I'd love to understand this further if y'all have any ideas.

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ This answer can be relevant. $\endgroup$
    – Alexey Popkov
    Oct 30, 2017 at 6:46
  • $\begingroup$ Can you tell us what headers you get? Especially content-type and content-encoding. $\endgroup$
    – Kuba ♦
    Oct 30, 2017 at 7:32
  • $\begingroup$ This is what I get from HTTPResponse[HTTPReq] "application/json;charset=utf-8". @Kuba. Do you need more info? $\endgroup$
    – Matt Green
    Oct 30, 2017 at 18:53
  • $\begingroup$ @MattGreen I guess the issue it the same as in linked topic. Nothing more to add here. But you can check if this isn't a faster workaround: ImportString[ToCharacterCode @ URLRead[..., "BodyBytes"], "JSON"] $\endgroup$
    – Kuba ♦
    Nov 2, 2017 at 7:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.