I'm trying to scrape this Wikipedia page ('Deaths in 2019- July', I'm writing this on the 14th.). I am failing miserably and I'm starting to think that it is due to some misconception on my part on the use of patterns.
Namely, the use of __
vs ..
and PatternSequence
.
Please help.
I use Import[<URL>, "XML"]
to import the page. I use pattn0
(see below) to identify the positions pos0 = Position[<XML>, pattn0]
and pos1 = MapAt[# + 1 &, pos0, {All, -1}]
.
pos
and pos1
are used to identify the positions of the desired entries.
I'm using src = Apply[Part[<XML>, ##] &, Riffle[pos0, pos1], 1]
in all pattern matching attempts as the source over which I try to extract (or match against) parts I want.
(I provide a link to the src
used, at the end of the question)
Snippet of desired entries
I will report below a random pair of entries I am interested in:
XMLElement["h3", {}, {XMLElement["span", {"class" -> "mw-headline", "id" -> "14"}, {"14"}]}],
XMLElement["ul",{},{
XMLElement["li",{},{
XMLElement["a",{"href"->"/wiki/Hussain_Muhammad_Ershad","title"->"Hussain Muhammad Ershad"},{"Hussain Muhammad Ershad"}],", 89, Bangladeshi military officer and politician,",
XMLElement["a",{"href"->"/wiki/Chief_of_Army_Staff_(Bangladesh)","title"->"Chief of Army Staff (Bangladesh)"},{"Chief of Army Staff"}],"(1978\[Dash]1986) and",
XMLElement["a",{"href"->"/wiki/President_of_Bangladesh","title"->"President of Bangladesh"},{"President"}],"(1983\[Dash]1990), myelodysplastic syndrome.",
XMLElement["sup",{"id"->"cite_ref-1","class"->"reference"},{XMLElement["a",{"href"->"#cite_note-1"},{"[1]"}]}]
}
],
XMLElement["li",{},{
XMLElement["a",{"href"->"/wiki/Nereo_Laroni","title"->"Nereo Laroni"},{"Nereo Laroni"}],", 76, Italian politician,",
XMLElement["a",{"href"->"/wiki/Mayor_of_Venice","title"->"Mayor of Venice"},{"Mayor of Venice"}],"(1985\[Dash]1987) and",
XMLElement["a",{"href"->"/wiki/Member_of_the_European_Parliament","title"->"Member of the European Parliament"},{"MEP"}],"(1989\[Dash]1994), complications from heart surgery.",
XMLElement["sup",{"id"->"cite_ref-2","class"->"reference"},{XMLElement["a",{"href"->"#cite_note-2"},{"[2]"}]}]
}
],
XMLElement["li",{},{
XMLElement["a",{"href"->"/wiki/Lavenia_Padarath","title"->"Lavenia Padarath"},{"Lavenia Padarath"}],", 74, Fijian politician,",
XMLElement["a",{"href"->"/wiki/Parliament_of_Fiji","title"->"Parliament of Fiji"},{"MP"}],"(1999\[Dash]2001, 2006) and President of the",
XMLElement["a",{"href"->"/wiki/Fiji_Labour_Party","title"->"Fiji Labour Party"},{"Labour Party"}],"(since 2015).",
XMLElement["sup",{"id"->"cite_ref-3","class"->"reference"},{XMLElement["a",{"href"->"#cite_note-3"},{"[3]"}]}]
}
]
}
]
Patterns I-am trying to-use
I am using the following patterns to match the entries I am interested in:
pattn0 = XMLElement["h3", _, {
XMLElement["span", {"class" -> "mw-headline", "id" -> id_}, {idnm_}]
}
]
(pattn0
matches against the day numbers reported in the header for each day)
and
pattn1 = XMLElement["a", {"href" -> href_, Longest[othr___], "title" -> tit_, last___}, {nm_}];
pattn2 = PatternSequence[pattn1, Longest[cmnt___]];
pattn2
is used to match against entries likeXMLElement["a", {__}, {_}]
which are the data entries I'm (mostly) interested in.- In
pattn2
I am usingLongest[othr___]
because there are some entries where, between"href"
and"title"
, there interjects some other minor entry (someXMLElement[__]
). - Also,
last___
is there to match against any remaining entries I might have missed. - Lastly,
Longest[cmnts___]
is used to match against entries (usually, as far as I can tell, a single one) afterXMLElement["a", __]
.
pattn3
and pattn4
match against reference entries:
pattn3 = XMLElement["a", {"href" -> href2_}, {refn_}];
pattn4 = XMLElement["sup", {"id" -> id_, "class" -> "reference", {pattn3}];
(pattn4
is used to match against XMLElement["sup", {__, {__}]
which are the entries related to references and their links)
pattn5
combines all the patterns into one (unfortunately non-performing) pattern:
pattn5 = PatternSequence[pattn0, x : XMLElement["ul", _, {XMLElement["li", _, {pattn2 .., pattn4}] ..}]];
Things that work
Retrieving day numbers Evaluating
Cases[src, pattn0 :> idnm]
works! and returns the day numbers{14,13,12,11,10,9,8,7,6,5,4,3,2,1}
as expected.
Retrieving footnotes/references
Both
pattn3
andpattn4
work. Here, evaluatingCases[src, pattn4, Infinity]
works (first five results):{ XMLElement[sup,{id->cite_ref-1,class->reference},{XMLElement[a,{href->#cite_note-1},{[1]}]}], XMLElement[sup,{id->cite_ref-2,class->reference},{XMLElement[a,{href->#cite_note-2},{[2]}]}], XMLElement[sup,{id->cite_ref-3,class->reference},{XMLElement[a,{href->#cite_note-3},{[3]}]}], XMLElement[sup,{id->cite_ref-4,class->reference},{XMLElement[a,{href->#cite_note-4},{[4]}]}], XMLElement[sup,{id->cite_ref-5,class->reference},{XMLElement[a,{href->#cite_note-5},{[5]}]}],...}
Things that partially work
pattn2
In using
pattn2
I get partially matched results; namely, the part related toLongest[cmnt___]
does not get matched, ever!. Essentially what works seems to be the part related topattn1
.Cases[src, pattn2, Infinity]
evaluates to the partially correct result (only the first five entries shown here to avoid clutter)
{ XMLElement[a,{href->/wiki/Hussain_Muhammad_Ershad,title->Hussain Muhammad Ershad},{Hussain Muhammad Ershad}], XMLElement[a,{href->/wiki/Chief_of_Army_Staff_(Bangladesh),title->Chief of Army Staff (Bangladesh)},{Chief of Army Staff}], XMLElement[a,{href->/wiki/President_of_Bangladesh,title->President of Bangladesh},{President}], XMLElement[a,{href->/wiki/Nereo_Laroni,title->Nereo Laroni},{Nereo Laroni}], XMLElement[a,{href->/wiki/Mayor_of_Venice,title->Mayor of Venice},{Mayor of Venice}], ...}
Things that should have(?) worked but don't!
Combined pattern
pattn5
Evaluating
Cases[src, pattn5, Infinity]
evaluates to{}
. Nothing gets matched!
Suggestions, Nudges, Tips on how to get a decent match
Are all welcome.
Thanks!
Appendix
I provide the src
used throughout the question (pastebin). Run Uncompress
on the string to retrieve the XML elements.
Uncompress["
then paste in the contents of the clipboard and continue typing without moving the cursor"]
4. evaluate $\endgroup$