13
$\begingroup$

Is it possible to use Mathematica to obtain just the title of a webpage? (I mean the small amount of text that appears in the webpage tab at the top of a browser.)

Every standard HTML page must have such a <title>HTML title tag</title> field in the <head>...</head> section.

I know that by using URLRead one can load all the information from a webpage. However, this function takes a few seconds to evaluate and fetch the data. So I wonder, in a case where we only want to know the page title, is it possible to request it directly without downloading other page data, and save time that way?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ I assume you're looking to check the title of many websites. You can speed this up by a lot if you download files asynchronously. Here is one example of that. Here is another. $\endgroup$
    – C. E.
    Feb 10, 2018 at 8:29
  • $\begingroup$ @C.E. Thank you for this useful suggestion! $\endgroup$
    – Kagaratsch
    Feb 10, 2018 at 14:40

3 Answers 3

Reset to default
14
$\begingroup$

As far as I am aware, HTTP doesn't support requesting the <title> element specifically. The browser can either GET the entire page, or request only the HTTP header information with HEAD (which notably does not include the HTML headers, where <title> is contained). It is also sometimes possible, as explained in WReach's answer, to request only specific byte ranges of the output page. For more information, see Mozilla's page on HTTP methods.

As the relevant standards don't have an implementation for this specifically, I would assume that there is no specific speed benefit to be had in trying to Import the title directly. Some simple testing agrees with this hypothesis on my machine:

AbsoluteTiming[Import["https://wolfram.com", "Title"]]

{1.51025, "Wolfram: Computation Meets Knowledge"}

AbsoluteTiming[URLRead["https://wolfram.com"];]

{1.47355, Null}

The actual difference in lookup times is negligible, and likely falls within random error on internet connection speeds and a little bit of string manipulation.

That said, kglr's suggestion of Import[...,"Title"] is an exceedingly useful syntax, and there's equally little reason not to use it and save yourself some string parsing.

Improve this answer
$\endgroup$
13
$\begingroup$

The HTTP protocol defines the Range header that can be used to limit the number of bytes returned in a response. Web sites are not required to support this header -- but if the target web site does then we may be able to use it to good effect.

We can tell if a web site claims to support the feature by looking for the Accept-Ranges response header from a HEAD request.

acceptRanges[url_] :=
  HTTPRequest[url, <|"Method"->"Head"|>] //
  URLRead[#, "Headers"]& //
  Lookup[#, "accept-ranges", "none"] &

Here are the results for some web sites:

{"www.imgur.com", "en.wikipedia.org", "www.wolfram.com"} //
AssociationMap[acceptRanges] //
Dataset

table showing acceptRanges results

We can define a function to retrieve the first bytes from a URL:

fetchFirstBytes[bytes_][url_] :=
  URLRead[
    HTTPRequest[url, <|"Headers" -> <|"Range" -> "bytes=0-"~~ToString[bytes-1]|>|>]
  , "Body"
  ]

For example:

$prefix = "https://www.imgur.com" // fetchFirstBytes[1024];

StringLength[$prefix]
(* 1024 *)

There is a good chance that the HTML title element will fall within the first kilobyte of the response (but adjust this guess to suit your taste). Since the content has been unceremoniously truncated, we cannot rely upon it containing well-formed HTML. We will have to extract the title using a string search:

$prefix // StringCases["<title>" ~~ title__ ~~ "</title>" :> title] // First // StringTrim
(* "Imgur: The magic of the Internet" *)

There is generally no harm in supplying the Range header unconditionally. If a web site does not support it, we will still get a response -- albeit the full content and no optimization benefit:

"www.wolfram.com" // fetchFirstBytes[1024] //
  StringCases["<title>"~~title__~~"</title>" :> title] // First // StringTrim

(* "Wolfram: Computation Meets Knowledge" *)

Beware that some sites will claim to support ranges but still return the entire response despite the presence of a Range header. There are a number of reasons for this such as content delivery networks, intermediate proxies or web application implementation details. So the bottom line is that we will need good luck to get the optimization. But we might as well try it and see.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Awesome piece of knowledge, thank you! Will try it out on a bunch of websites, and if it works to a benefit in most cases, then this answer will become the accepted one. $\endgroup$
    – Kagaratsch
    Feb 10, 2018 at 17:25
  • $\begingroup$ Very interesting! +1 $\endgroup$
    – C. E.
    Feb 11, 2018 at 7:06
11
$\begingroup$ 
Import["http://wolfram.com", "Title"]

"Wolfram: Computation Meets Knowledge"

Import["https://mathematica.stackexchange.com/questions", "Title"]

"Newest Questions - Mathematica Stack Exchange"

Improve this answer
$\endgroup$
4
  • $\begingroup$ +1, this will be convenient for OP. But until proven otherwise I will assume that it downloads the entire document and looks up the <title> tag. $\endgroup$
    – C. E.
    Feb 10, 2018 at 7:31
  • $\begingroup$ @C.E., may well be. Import["http://reference.wolfram.com/", "Elements"] gives "Title" as one of the elements , {"Data", "FullData", "Hyperlinks", "ImageLinks", "Images", "Plaintext", "Source", "Title", "XMLObject"}. Is "Title" is special, or does your comment applies to any of these elements? (Thank you for the upvote btw.) $\endgroup$
    – kglr
    Feb 10, 2018 at 7:41
  • $\begingroup$ I've done some web development and, as eyorble also points out, I don't think there is a way to request a part of a document with HTTP. Consequently, unless we're missing something all of those download the document in full. $\endgroup$
    – C. E.
    Feb 10, 2018 at 8:12
  • $\begingroup$ Thank you @C.E. Just saw eyorble's answer. $\endgroup$
    – kglr
    Feb 10, 2018 at 8:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.