1
$\begingroup$

I would like to webscrape college sports data. Specifically, I would like to get match by match data from baseball.

Manually, this is tiresome but possibly. I go to http://web1.ncaa.org/stats/StatsSrv/careersearch choose the school and the sport, click on the individual combination of school and year, and then click on the number right to "record". Finally, I have the data.

However, when using the Import command:

Import["http://web1.ncaa.org/stats/StatsSrv/careersearch", \
"Hyperlinks"]

Out: {"http://web1.ncaa.org/cgi-bin/ithelp/help.cgi?application=stats"}

I do not get further than the very first step. How can I get the links for the individual school and year combination?

All suggestions are welcome.

|improve this question
$\endgroup$
  • 1
    $\begingroup$ The navigation works by post requests. If you look at the HTML of each page, you will find a form (or several), demarcated by <form></form>. Navigation works by submitting those forms, either when the user clicks on a submit button or, via JavaScript, when he clicks on a link. You can see what parameters are sent along in fields that look like this: <input type="...">. Sometimes the fields are hidden, sometimes they correspond to form elements that you fill in yourself. $\endgroup$ – C. E. Sep 24 '19 at 19:52
  • 1
    $\begingroup$ You have to figure out what parameters you got to send and to what URL. On this website, it seems like the page you got to send the parameters to is the page you arrive at, so you can look at the HTML or simply the browser's address bar to figure out where to send the parameters. Then you gotta send the request by making the HTTPRequest object and sending it with resp = URLRead[request, "Body"]. Now, resp will be the HTML of that page. $\endgroup$ – C. E. Sep 24 '19 at 19:54
  • $\begingroup$ Thank you, now I am one step further. Lets see how I can progress from this point on. $\endgroup$ – Philippe Dufour Sep 25 '19 at 6:06
  • $\begingroup$ To parse the HTML, I recommend my jsoupLink package. Especially the element["Select", selector] syntax. That might help you programmatically find parameter values for your post request, f.e. I noticed that an intermediary page populates a hidden field with a number that depends on which team was selected on the previous page. $\endgroup$ – C. E. Sep 25 '19 at 7:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.