# Webscraping when links do not change

I would like to webscrape college sports data. Specifically, I would like to get match by match data from baseball.

Manually, this is tiresome but possibly. I go to http://web1.ncaa.org/stats/StatsSrv/careersearch choose the school and the sport, click on the individual combination of school and year, and then click on the number right to "record". Finally, I have the data.

However, when using the Import command:

Import["http://web1.ncaa.org/stats/StatsSrv/careersearch", \

• The navigation works by post requests. If you look at the HTML of each page, you will find a form (or several), demarcated by <form></form>. Navigation works by submitting those forms, either when the user clicks on a submit button or, via JavaScript, when he clicks on a link. You can see what parameters are sent along in fields that look like this: <input type="...">. Sometimes the fields are hidden, sometimes they correspond to form elements that you fill in yourself. – C. E. Sep 24 '19 at 19:52
• You have to figure out what parameters you got to send and to what URL. On this website, it seems like the page you got to send the parameters to is the page you arrive at, so you can look at the HTML or simply the browser's address bar to figure out where to send the parameters. Then you gotta send the request by making the HTTPRequest object and sending it with resp = URLRead[request, "Body"]. Now, resp will be the HTML of that page. – C. E. Sep 24 '19 at 19:54
• To parse the HTML, I recommend my jsoupLink package. Especially the element["Select", selector] syntax. That might help you programmatically find parameter values for your post request, f.e. I noticed that an intermediary page populates a hidden field with a number that depends on which team was selected on the previous page. – C. E. Sep 25 '19 at 7:17