7
$\begingroup$

In version 11,if we want to crawl public website like wiki,we can do it like this:

URLRead["https://en.wikipedia.org/wiki/Wiki", "Body"]

But I don't know how to crawl some website needed username and password,such as Stack Exchange,our Facebook homepage or our educational management system.Of course some webs providing some API can help us to do this.but I don't want to use method of API to do this.Anyone can give a example to do it?For example,how to crewl the Stack Exchange of Mathematica.I want to get all of the favorite items and its number of vote in my homepage like following

I prefer to the answer use Mathemtica of version 11,because it start to support cookie(FindCookies),which maybe help to simplify the process.

Improve this question
$\endgroup$
8
  • 6
    $\begingroup$ The usual way is to look at the login form and see where it sends there login data, then send login data to that address. The website will return a cookie, and this cookie will be used in future communication with the website to show that you are logged in. Now, websites like Facebook and StackExchange are sure to look down on users that do this; they provide APIs for a reason. There is information they want to expose, and information they don't want to expose. Therefore most sites will try to prevent you from doing this type of scraping, which means it turns into a game of cat and mouse. $\endgroup$
    – C. E. ♦
    Commented Aug 12, 2016 at 12:00
  • $\begingroup$ So, you have to set up your requests in such a way that it looks like it's coming from a web browser. You have to trick Facebook that your Mathematica script is actually a human. If Facebook just slightly suspects that you are not a human, it will ask you to fill out a reCaptcha, which are built specifically to be difficult to bypass programmatically. It's difficult to fool well protected websites, and it's all wholly unrelated to Mathematica. You could ask on StackOverflow first, and then come back and ask how to implement what you found in Mathematica, for example. That would be better. $\endgroup$
    – C. E. ♦
    Commented Aug 12, 2016 at 12:03
  • $\begingroup$ @C.E. Thanks for you information,which helpful to me. :) $\endgroup$
    – yode
    Commented Aug 12, 2016 at 12:18
  • $\begingroup$ I've done this several years ago using PHP and cURL, but never aimed for websites like Facebook. If you set your bar lower, then you might be able to get by with changing the HTTP header that specifies the user agent to some browser. It is very common to provide a whitelist of allowed user agents (user agents are f.e. browsers). $\endgroup$
    – C. E. ♦
    Commented Aug 12, 2016 at 13:02
  • $\begingroup$ @C.E. Wow,can you give a simple example by Mthematica as you say that?And sorry that I don't know how difficult crawl SE. $\endgroup$
    – yode
    Commented Aug 12, 2016 at 13:05

0

Reset to default

Browse other questions tagged or ask your own question.