Parsing JavaScript Script from XML Trees

Question

I've been trying to parse some script written in JavaScript while scraping a website, and am a little confused what functions are needed to get this done in Mathematica.

I imported the data using Import["url","XMLObject"]. The website I am trying to scrape is https://www.msn.com/en-us/money/stockdetails/history/fi-126.1.AAPL.NAS. I am basically trying to scrape historical data prices of certain dates, all of which are hidden in javascript. Any help would be of great appreciation!

The XML for the desired part looks like this:

XMLElement["script", {"type" -> 
   "text/javascript"}, {"\n    require([\"jqLoadTemplate\", \
\"dateFormat\", \"ChartsConfig\", \"chartUtils\", \
\"ChartModeController\", \"utils\", \"LocaleSettings\", \
\"NumberFormatter\", \"c.deferred\"], function ($, dateFormat, \
chartsConfig, chartUtils, ChartModeController, commonUtils, settings, \
numberFormatter) {\n        var configObject = \
chartsConfig[\"StockDetails\"];\n        var chartOptions = \
configObject.chartOptions || {};\n        if (\"False\" === \"True\") \
{\n            configObject.defaultDateRange = \
configObject.defaultOtcRange;\n        }\n\n        function \
loadTable(data) {\n            // Create a table\n            if \
(data != null && data[0] && data[0].seriesId && data[0].series) {\n   \
             var historicalData = [];\n                var size = \
data[0].series.length;\n                var formatString;\n           \
     switch (data[0].Ct) {\n                    case \"1D\":\n        \
            case \"1D_5M\":\n                        formatString = \
settings.shortTimePattern;\n                        break;\n          \
          case \"5D\":\n                        formatString = \
settings.shortTimePattern + \" \" + settings.dayDatePattern;\n        \
                break;\n                    default:\n                \
        formatString = settings.monthDayYearPattern;\n                \
}\n\n                var settingsCopy = $.extend(true, {}, settings);\
\n                settingsCopy.numberDecimalDigits = 0;\n\n           \
     for (var j in data[0].series) {\n                    i = \
data[0].series[size - 1 - j];\n                    var date = \
dateFormat(chartUtils.normalizeDate(i[0]), formatString);\n           \
         var open = numberFormatter.formatNumber(i[4], settings);\n   \
                 var close = numberFormatter.formatNumber(i[1], \
settings);\n                    var low = \
numberFormatter.formatNumber(i[6], settings);\n                    \
var high = numberFormatter.formatNumber(i[5], settings);\n            \
        var volume = numberFormatter.formatNumber(i[3], \
settingsCopy);\n                    historicalData.push({\n           \
             \"date\": date,\n                        \"high\": high,\
\n                        \"low\": low,\n                        \
\"open\": open,\n                        \"close\": close,\n          \
              \"vol\": volume\n                    });\n              \
  }\n                \
$('#containerTable').loadTemplate(('#trTemplate'), historicalData);\n \
           }\n\n        }\n        //dataUrl is URL template. \
urlResolver inserts the parameters.\n        function \
onModeChanged(timeRangeSelected, chartTypeSelected) {\n            \
var dataUrl = configObject.getUrl(timeRangeSelected, \"False\", \
\"False\");\n            var urlResolver = configObject.urlResolver;\n\
            var finalUrl = urlResolver(dataUrl, { symbol: \
\"126.1.AAPL.NAS\", isEOD: \"False\", locale: \"en-US\", \
timeRangeSelected: timeRangeSelected, chartTypeSelected: \"ohlc\", \
isVolumeChartSupported: true });\n            \
chartUtils.getData(finalUrl, configObject.dataBuilder, loadTable, \
function () { }, \"ohlc\");\n        }\n        var timeRange = \
configObject.getTimeRange(\"False\", \"False\", \"126.1.AAPL.NAS\");\n\
            var control = \
ChartModeController.initialize(configObject, onModeChanged, \
\"False\", {}, \"126.1.AAPL.NAS\", timeRange);\n            var \
modeSelector = \
document.getElementById(\"heropanechart-mode-selector\");\n        \
modeSelector.appendChild(control);\n        onModeChanged(timeRange, \
configObject.getDefaultChartType());\n    });\n    "}], "\n ",

Answer 1

You have misunderstood how this works. What happens is that JavaScript calls a server-side script which returns the data, and the data is subsequently inserted into the HTML. The data cannot be found in the HTML document (in the HTML or in the JavaScript) to begin with.

The most straightforward way to get the data is to monitor the network and observe what URLs the JavaScript code is calling. There is a network tab in Chrome's web developer tool that will do this for you. I opened up the network tab, clicked the "WEEK" tab on the website and took notice of what URL was added to the list of called URLs. It turned out to be this one:

url = "https://finance.services.appex.bing.com/Market.svc/ChartDataV5?symbols=126.1.AAPL.NAS&chartType=5d&isEOD=False&lang=en-US&isCS=true&isVol=true";

The format of the data is JSON. We can import it like this:

data = Import[url, "JSON"];
values = "Series" /. data;

values is now a list of elements such as this:

{"T" -> 10260, "P" -> 141.23, "Hp" -> 141.25, "Lp" -> 141.14, 
 "Op" -> 141.14, "V" -> 34837}

It is unclear what timestamp format is being used, but Hp is definitely the high, Lp the low, Op the open value, p the close value, and V the volume.

To get data for other stocks you have to make the appropriate update to the URL. Changing AAPL.NAS (Apple) to MSFT.NAS (Microsoft) seems to work.

Of course, this is not how the website is meant to be used. They may change the URLs in whatever way they chose, at any moment in time. But this approach may very well work for downloading the data that you want.