I'm trying to import all the tables on a website, this one in particular:
For local files that are already tabular, Import[#, "Table"] works well, but I can't seem to easily import simple tables embbeded in a sites.
2
$\begingroup$
$\endgroup$
-
$\begingroup$ At first glance, you may need to look outside of Mathematica for this one. It looks like the prices are generated with JS, onload. Selenium makes a Java package you could most likely wrap into Mathematica. $\endgroup$ – Peter Roberge Oct 5 '15 at 18:17
-
$\begingroup$ If you know the basics of CSS selectors and HTML, then you can download the website after the prices have been loaded and use jSoupLink on your downloaded files. $\endgroup$ – C. E. Oct 5 '15 at 18:34
-
$\begingroup$ @Pickett that's a good idea! $\endgroup$ – M.R. Oct 5 '15 at 20:23
-
$\begingroup$ @PeterRoberge I'm looking into other platforms... but I would think URLFetchAsynchronous[] could handle this? $\endgroup$ – M.R. Oct 5 '15 at 20:25
-
1$\begingroup$ Amazon loads a JavaScript file which contains the data in JSON format with a JavaScript 'callback'. For example for Linux the file is a0.awsstatic.com/pricing/1/ec2/linux-od.min.js You could parse this file and the other files similarly to get the data. $\endgroup$ – user9444 Oct 6 '15 at 23:53
1
$\begingroup$
$\endgroup$
This is only a partial answer because it turns out the AWS JSON is surprisingly complicated. Anyways this might help you get there eventually. Lots of work left unfortunately.
str = Import["http://a0.awsstatic.com/pricing/1/ec2/linux-od.min.js", "String"];
pos = StringPosition[str, "callback"];
obj = StringTake[str, {pos[[1, 2]] + 2, StringLength[str] - 2}];
json = StringReplace[obj, ":" -> "->"];
json = StringReplace[json, "{" -> "<|"];
json = StringReplace[json, "}" -> "|>"];
json = StringReplace[json, "[" -> "{"];
json = StringReplace[json, "]" -> "}"];
json = ToExpression[json];
- Import the JavaScript file pointed to on the webpage that contains the data.
- Extract the JSON part (getting rid of JavaScript stuff, comments, etc)
- String replace JSON parts to Mathematica equivalents and to turn this into a Mathematica Association with ToExpression.
Now we have an association. We can now access the data in a hierarchical manner e.g.
json[config][regions][[1]][instanceTypes][[1]][sizes][[1]]
which gives us the result:
<|size -> "t2.micro", vCPU -> "1", ECU -> "variable",
memoryGiB -> "1", storageGB -> "ebsonly",
valueColumns -> {<|name -> "linux", prices -> <|USD -> "0.013"|>|>}|>
