2
$\begingroup$

The URLFetch command is very useful for pulling the entire html source of a given webpage, however, if one wanted to speed up the process, by only requesting a portion of the page, for instance, only the body, how would one do this? A given page of html might be quite large, and querying several pages in sequence could slow the process down significantly, if all one really wants is a small portion of that page.

At present, it is possible to specify: "Content", "ContentData", "Headers", "Cookies", "StatusCode". I thought that "Content" would pull only the body portion, excluding the headers, but this is not the case.

Alternatively, is it possible to request a specific line or lines from within the html code?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ The size of the headers is very small compared to the time it takes to establish a connection with the server. I don't think there is anything you can do to reduce the size of the request enough for it to matter, but you might be interested in URLFetchAsynchrounous which allows you to download many pages simultaneously. $\endgroup$ – C. E. Feb 11 '15 at 20:19
  • $\begingroup$ @Pickett Fair point, what are your thoughts on the second part of my question, that is trying to extract a specific line or lines from the html code? URLFetchAsynchronous I agree is much faster, and I'd prefer to use it, however, I am pulling pages based on the results of the prior page, so it appears to be inherently sequential. $\endgroup$ – iwantmyphd Feb 11 '15 at 20:28
  • $\begingroup$ AFAIK it is not possible, unfortunately. $\endgroup$ – C. E. Feb 11 '15 at 21:30
  • $\begingroup$ Before URLFetch I used to use wget regularly. You might want to consider trying it if you are having problems with URLFetch $\endgroup$ – Mike Honeychurch Feb 11 '15 at 21:46
  • $\begingroup$ @MikeHoneychurch How do you call wget from Mathematica? $\endgroup$ – iwantmyphd Feb 11 '15 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.