10
$\begingroup$

How can I save all citations from a Google Scholar search? For instance, in a search for "Radon transformation", there are about 35,500 results. I want to download all citations into a string - is there a simple way to get Mathematica to do that? (And sorry, no, I haven't tried anything yet. Not sure where to start.)

$\endgroup$
6
  • 1
    $\begingroup$ That surely would be against the fair use policy, wouldn't it? $\endgroup$
    – rhermans
    Oct 13, 2015 at 17:39
  • $\begingroup$ @rhermans I don't know - I haven't checked that. I am certainly not intentionally trying to do something that breaks their rules - I figured it would be fair to try. If it is against fair use, then never mind. $\endgroup$
    – GregH
    Oct 13, 2015 at 17:47
  • $\begingroup$ A few tens of queries surely are fine but 35K will make you noticed. $\endgroup$
    – rhermans
    Oct 13, 2015 at 17:48
  • $\begingroup$ The policy states "Don’t misuse our Services. For example, don’t interfere with our Services or try to access them using a method other than the interface and the instructions that we provide." I was hoping to use MMA to automate the "interface interaction" process, but maybe that violates the policy. $\endgroup$
    – GregH
    Oct 13, 2015 at 18:11
  • 6
    $\begingroup$ @rhermans, In September 2014, the SDNY upheld TVEye’s affirmative defense of fair use with respect to its core function of indexing news clips in a searchable database. The court reasoned that this indexing function is transformative because it “converts copyrighted works into a research tool to further learning.” $\endgroup$ Oct 13, 2015 at 20:52

2 Answers 2

7
$\begingroup$

This could get you strated

gsq[str_String] := Block[{url},
  url = "https://scholar.google.co.uk/scholar?hl=en&q=" <> 
    URLEncode[str] <> "&btnG=&as_sdt=1%2C5&as_sdtp=";
  Cases[
   Cases[
    Import[url, "XMLObject"]
    , XMLElement["div", {"id" -> "gs_ccl", "role" -> "main"}, ___], 
    Infinity]
   , XMLElement["h3", {"class" -> "gs_rt"}, ___], Infinity]
  ]

Use "inspect element" in your browser to understand the div ids.

$\endgroup$
6
$\begingroup$

(This is an abbreviated version of a post on my blog

Google Scholar (via scholarly)

GoogleScholar is not really intended for programmatic access, and you may also hit usage limits. That being said, python packages like scholarly exist to do this, and if you make only a few requests, you should be fine:

(* install the python package *)
ResourceFunction["PythonPackageInstall"]["scholarly"];

(* create a python session and function to get the first 10 results *)
session = StartExternalSession[
    <|"System" -> "Python", 
     "SessionProlog" -> "from scholarly import scholarly"|>] ;
 
searchPubs = ExternalFunction[session, 
  "def search(terms):
    query = scholarly.search_pubs(terms)
    results = []
    for i in range(10):
      results.append(next(query))
    return results"];

(* demo: perform the search *) 
example = searchPubs["Radon transform"];

(* extract information from the result *)
Dataset @ Query[All, "bib", {"pub_year", "author"}] @ example

A Better Way: OpenAlex API

OpenAlex is an open project that makes it easy to get data. The easiest way is get started is to do a text search through the web, and then click the gear button to generate the corresponding API query:

screen snapshot of an OpenAlex web search

It is then straightforward to copy and execute the corresponding API query in using URLExecute (you may of course modify it to suit your needs:

example2 = URLExecute["https://api.openalex.org/works?page=1&filter=default.search:radon+transform&sort=relevance_score:desc&per_page=10"]; 
 
Dataset @ Query["results", All, {"publication_date", "doi"}] @ example2

OpenAlex has many other powerful features, including (machine-learned) topic groupings, links to open access versions of documents, etc. The documentation is excellent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.