I would like to webscrape college sports data. Specifically, I would like to get match by match data from baseball.
Manually, this is tiresome but possibly. I go to http://web1.ncaa.org/stats/StatsSrv/careersearch choose the school and the sport, click on the individual combination of school and year, and then click on the number right to "record". Finally, I have the data.
However, when using the Import command:
Import["http://web1.ncaa.org/stats/StatsSrv/careersearch", \
"Hyperlinks"]
Out: {"http://web1.ncaa.org/cgi-bin/ithelp/help.cgi?application=stats"}
I do not get further than the very first step. How can I get the links for the individual school and year combination?
All suggestions are welcome.
<form></form>
. Navigation works by submitting those forms, either when the user clicks on a submit button or, via JavaScript, when he clicks on a link. You can see what parameters are sent along in fields that look like this:<input type="...">
. Sometimes the fields are hidden, sometimes they correspond to form elements that you fill in yourself. $\endgroup$ – C. E. Sep 24 at 19:52HTTPRequest
object and sending it withresp = URLRead[request, "Body"]
. Now,resp
will be the HTML of that page. $\endgroup$ – C. E. Sep 24 at 19:54element["Select", selector]
syntax. That might help you programmatically find parameter values for your post request, f.e. I noticed that an intermediary page populates a hidden field with a number that depends on which team was selected on the previous page. $\endgroup$ – C. E. Sep 25 at 7:17