The URLFetch
command is very useful for pulling the entire html source of a given webpage, however, if one wanted to speed up the process, by only requesting a portion of the page, for instance, only the body, how would one do this? A given page of html might be quite large, and querying several pages in sequence could slow the process down significantly, if all one really wants is a small portion of that page.
At present, it is possible to specify: "Content"
, "ContentData"
, "Headers"
, "Cookies"
, "StatusCode"
. I thought that "Content"
would pull only the body portion, excluding the headers, but this is not the case.
Alternatively, is it possible to request a specific line or lines from within the html code?
URLFetchAsynchrounous
which allows you to download many pages simultaneously. $\endgroup$ – C. E. Feb 11 '15 at 20:19URLFetchAsynchronous
I agree is much faster, and I'd prefer to use it, however, I am pulling pages based on the results of the prior page, so it appears to be inherently sequential. $\endgroup$ – iwantmyphd Feb 11 '15 at 20:28URLFetch
I used to use wget regularly. You might want to consider trying it if you are having problems withURLFetch
$\endgroup$ – Mike Honeychurch Feb 11 '15 at 21:46wget
from Mathematica? $\endgroup$ – iwantmyphd Feb 11 '15 at 21:50