# Scraping 2020 Polling Data w/Mathematica?

Is there a way to use Mathematica to scrub the latest ("RCP average") polling data for each of the 2020 Democratic Primary candidates?

Pretty simple using Import, select the section you want, then turn it into a dataset.

data = Import[
"https://www.realclearpolitics.com/epolls/2020/president/us/2020_\
democratic_presidential_nomination-6730.html#polls", "Data"][[2]];
ds = Dataset[AssociationThread[First[data] -> #] & /@ Rest[data]]


Luckily this website is structured in a way that Mathematica interprets easily. It can be a bit trickier when they use a lot of AJAX.

Once you have the data in, there are quite a few things you can do with it.

For instance, if you want to call Biden's most recent poll:

pollsonly = Rest[ds];
biden = pollsonly[[1, "Biden"]]


Or if you wanted the average of his last three polls:

bidenlast3 = pollsonly[[1 ;; 3, "Biden"]] // Mean // N


Since different polls are completed at different intervals, you might not just want the most recent polls overall, but rather the most recent poll from every pollster who has completed a poll in the last 30 days. This code would get you that:

recentpolls =
Select[pollsonly[
All, {"Date" -> (Quiet[
DateObject[Last[StringSplit[#, " - "]]]] &)}],
QuantityMagnitude[Today - #Date] < 30 &];
Values[First /@ GroupBy[recentpolls, "Poll"]][[All, "Biden"]] //
Mean // N


You could also look at movement of the candidates across a single poll:

politicopolls =
Select[pollsonly, #[[1]] == "Politico/Morning Consult Politico" &];
ListLinePlot[
Merge[Normal[Reverse[politicopolls[[All, 3 ;; -2]]]], Identity] /.
"--" -> 0, PlotLabels -> Automatic, PlotLegends -> None,
PlotRange -> {All,All}, ImageSize -> Large]


Lots of fun things to do. Enjoy!

• Is the best way to programatically grab, i.e., Biden's latest poll average ds[2, "Biden"]? Is there a way to use "RCP Average" instead of "2" (just in case the rows switch around on future API grabs)? Commented Jul 1, 2019 at 18:45
• @George. Just updated my answer to give you a few more options. Commented Jul 2, 2019 at 13:03