This question already has an answer here:

I am trying to use Mathematica to analyze the data given by the Boston Marathon concerning running time, etc. The website http://registration.baa.org/2015/cf/Public/iframe_ResultsSearch.cfm gives a form for requesting data. One can get a lot by just selecting the Gender to be either M or F and requesting 1000 results. However, the data comes in just groups of 25. How can I use Mathematica to get all 1000 results without manually getting the data in groups of 25? Better yet ... how can I get info on all the runners, not just the top 1000?

I know this is not necessarily a Mathematica question, but Mathematica is my favorite tool for this kind of thing and I'm trying to figure out how to use it.

What I've done so far: I can run a search - say, for Males and request 1000 results. When the first 25 results come up, I can view the page source and near the bottom their is a line that begins with <input type="hidden" name="start" value="26"/>. By changing the value from 26 to say, 27, and saving the results as a webpage, I can see the results numbered 27 to 51 when I click on the "Next 25 Records" button. I don't know how to get Mathematica to submit this form for me, nor click through the "next results". This may just be a poor way of approaching the problem, and a more elegant method exists.


marked as duplicate by Jens, MarcoB, m_goldberg, Oleksandr R., Dr. belisarius Aug 20 '15 at 6:50

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ If there's no public API for automated requests, this is basically a question about hacking their form submission interface to scrape the data. Therefore, the main issue is really quite unrelated to Mathematica. Have you tried anything? $\endgroup$ – Jens Aug 19 '15 at 18:01
  • $\begingroup$ Yes, I did, though didn't describe anything above. And yes, I just tried to hack their submission form to get more than 25 results at a time to no avail. I did figure a way to change the start value of subsequent pages of results - so instead of seeing 1-25, then 26-50, etc., I can view 27-51 if I desire. But I did that with manual hacking, and haven't yet figured out how to get Mathematica to do that for me. I don't want this to be a case of "Solve my problem for me," but many on this forum seem to know a lot about a lot of things and hoped someone could point me in the right direction. $\endgroup$ – GregH Aug 19 '15 at 18:10
  • 1
    $\begingroup$ Perhaps you can add details of what you have already done manually, if you think it can be automated. $\endgroup$ – Jens Aug 19 '15 at 18:29
  • $\begingroup$ I've added some details above. Thanks. $\endgroup$ – GregH Aug 19 '15 at 18:44
  • 1
    $\begingroup$ This question is about web scraping. $\endgroup$ – Enrique Pérez Herrero Aug 19 '15 at 18:50

Browse other questions tagged or ask your own question.