I am trying to extract bibliographies from academic papers which are in pdf format, and I have to do this for 100+ papers. My goal is to create a network of citations amongst the authors and identify the communities. Is there a way to do this with Mathematica instead of manually copying and pasting bibliographies, paper-by-paper?
$\begingroup$
$\endgroup$
2
-
2$\begingroup$ Many journals already provide the list of references (in html format) on the web page for the article. Some also give a list of papers that cite the article in question. $\endgroup$– Simon RochesterOct 9, 2015 at 0:08
-
1$\begingroup$ check out Mendalay and Docear both programs will get bibliography data from files. $\endgroup$– user34696Oct 9, 2015 at 0:40
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
This is actually a complex problem because different journals and papers have different names for the bibliography (e.g. "Bibliography" versus "References") and different referencing formats that are hard to parse. Also some papers put other material (e.g. appendices) after the bibliography rather than before.
That said, here is something to get you started. I am using a working paper I happened to have lying around called "Clustering compositional data trajectories" by F Bruno and F Greco from the University of Bologna.
As long as the PDF is not password-protected you can extract the text like this:
Last[StringSplit[
Import["BrunoGreco.pdf", {"PDF", "Plaintext"}], {"References", "Bibliography"}]]
Specifying both tags means you do not have to check whether each paper uses "References" or "Bibliography", but there is a risk that someone uses the word "References" in the main text. Picking the Last of the split strings will deal with that. You will get output like this:
Abraham, C., Cornillon, P.A., Matzner-Løber, E., Molinari, N. (2003). \
Unsupervised Curve Clustering
using B-Splines. Scandinavian Journal of Statistics 30(3), 581-595.
Aitchison, J. (1982). The statistical analysis of compositional data \
(with discussion). J. R. Statist. Soc. B.,
44, 139-177.
Parsing the bibliography items is a separate issue and will depend on the format of the bibliographic entries.
Edit
Using Sjoerd's test file, I came up with the following partial solution:
test1 = FixedPoint[
StringReplace[#, {"\n" ~~ Whitespace ~~ "\n" -> "\n",
" " -> ""}] & /@ # &, (StringTrim[#] & /@ (StringReplace[#, \
{"[" ~~ DigitCharacter .. -> "", "\[RawEscape]" -> "ff",
"\.15" -> "\[Dash]"}] & /@
StringSplit[
Last[StringSplit[
Import["13092561v2.pdf", {"PDF",
"Plaintext"}], {"References", "Bibliography"}]], "]"]))]
{"", "J.W.Burby,J.Squire,H.Qin,Automationoftheguidingc enterexpansion, PhysicsofPlasmas20(7)(2013)072105.", \ "Mathematica,Version9.0,WolframResearchInc.,Champ \ aign,Illinois,2012.", \ "J.W.Eastwood,ORTHOVEC:version2oftheREDUCEprogram for3-D
vectoranalysisinorthogonalcurvilinearcoordinates,Co mputerPhysicsCom- munications64(1)(1991)121[Dash]122.", \ "P.B.Yasskin,A.Belmonte,CalcLabswithMaplecommandsformultivariable
calculus,fourthedition,Brooks/Cole,CengageLearning, \ Belmont,CA,2010.", "B.Fiedler,Vectan1.1,manualMath.Inst.,Univ.Leipzi \ g.(1997).", \ "M.C.Wirth,Symbolicvectoranddyadicanalysis,SiamJ.Comput.8(3)(1979)306.", "S.Liang,D.J.Jeffrey,Rule-basedsimpli.1ccationinvect \ or-productspaces,To-
wardsMechanizedMathematicalAssistants,LectureNotesinArti.1c\ cialIntelli- gence(2007)116[Dash]127. "
This cleans up some of the text and deals with a few encoding problems. It should be enough to get you started. You probably only need to parse the authors not the rest of the bibliographic information, so splitting strings at dates and then parsing the first bit at commas should work.
Here is an example that works for the initials-dot-surname format of the second paper. It doesn't pick up authors like "Mathematica", which is in this bibliography list, but I suspect this is not fatal to your application.
test2 = (Select[#, (StringMatchQ[#, (LetterCharacter ~~
".") .. ~~ (LetterCharacter |
"-") ..]) &] & /@ (StringSplit[#, ","] & /@ test1)) /. {} -> Sequence[]
(*{{"J.W.Burby", "J.Squire", "H.Qin"}, {"J.W.Eastwood"}, {"P.B.Yasskin",
"A.Belmonte"}, {"B.Fiedler"}, {"M.C.Wirth"}, {"S.Liang",
"D.J.Jeffrey"}, {"H.Qin", "W.Tang",
"G.Rewoldt"}, {"J.R.Cary"}, {"J.R.Cary",
"A.J.Brizard"}, {"R.G.Littlejohn"}, {"J.D.Huba"}, {"Y.Wang"}, \
{"L.Parker",
"S.Christensen"}, {"K.Peeters"}, {"J.M.Martín-García"}, \
{"D.Stoutemyer"}, {"J.M.Martín-García"}, {"L.R.U.Manssur",
"R.Portugal",
"B.Svaiter"}, {"J.Cunningham"}, {"E.Patterson"}, {"H.K.Wimmel"}, \
{"R.G.Littlejohn"}, {"D.Lovelock"}, {"S.B.Edgar",
"A.Hoglund"}, {"J.M.Martín-García", "R.Portugal", "L.Manssur"}} *)
You can then get a mini graph like this (Squire, Burby and Qin being the authors of that paper):
links = Flatten@ Outer[#1 -> #2 &, {"J.Squire", "J.Burby", "H.Qin"},
Union@Flatten@test2]
(*{"J.Squire" -> "A.Belmonte", "J.Squire" -> "A.Hoglund",
"J.Squire" -> "A.J.Brizard", "J.Squire" -> "B.Fiedler",
"J.Squire" -> "B.Svaiter", "J.Squire" -> "D.J.Jeffrey",
"J.Squire" -> "D.Lovelock", "J.Squire" -> "D.Stoutemyer",
"J.Squire" -> "E.Patterson", "J.Squire" -> "G.Rewoldt",
"J.Squire" -> "H.K.Wimmel", "J.Squire" -> "H.Qin",
"J.Squire" -> "J.Cunningham", "J.Squire" -> "J.D.Huba",
"J.Squire" -> "J.M.Martín-García", "J.Squire" -> "J.R.Cary",... etc *)
Graph[links]
-
$\begingroup$ I was working with a different test file ("arxiv.org/pdf/1309.2561v2.pdf") using similar methods, but was less lucky. PDFs can be a mess. $\endgroup$ Oct 8, 2015 at 22:37
-
$\begingroup$ Thank you so much for the help!!! I'm going to try it right now!! $\endgroup$– pollonOct 9, 2015 at 9:44
$\begingroup$
$\endgroup$
I would use the approach suggested by Simon Rochester above. Just do enough PDF parsing to extract the DOI or URL of the web page for the paper, then use that information to either extract 'Cited by' information directly from the paper's publisher web page, or get the same information by passing the paper's details to Crossref, Google Scholar or one of the other academic databases which keeps track of 'Cited by' information.
