2
$\begingroup$

I'm trying to import all the tables on a website, this one in particular:

For local files that are already tabular, Import[#, "Table"] works well, but I can't seem to easily import simple tables embbeded in a sites.

| improve this question | |
$\endgroup$
  • $\begingroup$ At first glance, you may need to look outside of Mathematica for this one. It looks like the prices are generated with JS, onload. Selenium makes a Java package you could most likely wrap into Mathematica. $\endgroup$ – Peter Roberge Oct 5 '15 at 18:17
  • $\begingroup$ If you know the basics of CSS selectors and HTML, then you can download the website after the prices have been loaded and use jSoupLink on your downloaded files. $\endgroup$ – C. E. Oct 5 '15 at 18:34
  • $\begingroup$ @Pickett that's a good idea! $\endgroup$ – M.R. Oct 5 '15 at 20:23
  • $\begingroup$ @PeterRoberge I'm looking into other platforms... but I would think URLFetchAsynchronous[] could handle this? $\endgroup$ – M.R. Oct 5 '15 at 20:25
  • 1
    $\begingroup$ Amazon loads a JavaScript file which contains the data in JSON format with a JavaScript 'callback'. For example for Linux the file is a0.awsstatic.com/pricing/1/ec2/linux-od.min.js You could parse this file and the other files similarly to get the data. $\endgroup$ – user9444 Oct 6 '15 at 23:53
1
$\begingroup$

This is only a partial answer because it turns out the AWS JSON is surprisingly complicated. Anyways this might help you get there eventually. Lots of work left unfortunately.

str = Import["http://a0.awsstatic.com/pricing/1/ec2/linux-od.min.js", "String"];
pos = StringPosition[str, "callback"];
obj = StringTake[str, {pos[[1, 2]] + 2, StringLength[str] - 2}];
json = StringReplace[obj, ":" -> "->"];
json = StringReplace[json, "{" -> "<|"];
json = StringReplace[json, "}" -> "|>"];
json = StringReplace[json, "[" -> "{"];
json = StringReplace[json, "]" -> "}"];
json = ToExpression[json];
  • Import the JavaScript file pointed to on the webpage that contains the data.
  • Extract the JSON part (getting rid of JavaScript stuff, comments, etc)
  • String replace JSON parts to Mathematica equivalents and to turn this into a Mathematica Association with ToExpression.

Now we have an association. We can now access the data in a hierarchical manner e.g.

json[config][regions][[1]][instanceTypes][[1]][sizes][[1]]

which gives us the result:

<|size -> "t2.micro", vCPU -> "1", ECU -> "variable",
memoryGiB -> "1", storageGB -> "ebsonly",
valueColumns -> {<|name -> "linux", prices -> <|USD -> "0.013"|>|>}|>
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.