4
$\begingroup$

We wish to analyze the tree structure of HTML documents that consist of nested lists, ie, using the HTML tags <ul> and <li>. The documents are generated by Workflowy, a note taking web app that can manage arbitrarily deep nesting (and dynamically re-arrange them by zooming and searching).

For example, here's a portion of one document:

<ul class="ul1">
  <li class="li2">Subjective</li>
  <ul class="ul1">
    <li class="li2">XX yr old man here for fu on med problems</li>
    <li class="li2">feels absolutely positively wonderful</li>
  </ul>
  <li class="li2">Active problems</li>
  <ul class="ul1">
    <li class="li2">htn</li>
    <ul class="ul1">
      <li class="li2">started on hctz in 8/10, in early 11 given lasix as his gfr was low</li>
      <li class="li2">limits salt caffeine, home readings140/80  or less when he runs</li>
      <li class="li2">not been exercising now</li>
    </ul>

After applying the main parse function in Leonid Shifrin's HTML parser (Thanks for that), the above portion is transcoded to:

ulContainer[attribContainer[ class="ul1"],
  ,liContainer[attribContainer[ class="li2"],Subjective],
  ,ulContainer[attribContainer[ class="ul1"],
    ,liContainer[attribContainer[ class="li2"],XX yr old man here for fu on med problems],
    ,liContainer[attribContainer[ class="li2"],feels absolutely positively wonderful],
  ],
  ,liContainer[attribContainer[ class="li2"],Active problems],
  ,ulContainer[attribContainer[ class="ul1"],
    ,liContainer[attribContainer[ class="li2"],htn],
    ,ulContainer[attribContainer[ class="ul1"],
      ,liContainer[attribContainer[ class="li2"],started on hctz in 8/10, in early 11 given lasix as his gfr was low],
      ,liContainer[attribContainer[ class="li2"],limits salt caffeine, home readings140/80  or less when he runs],
      ,liContainer[attribContainer[ class="li2"],not been exercising now],
    ],

Shifrin writes:

What it gives you is a symbolic tree representation of the parsed html document, with different tags transformed into ...Container[content], for example divContainer[aContainer["Some text"]]. The container names mirror the html tag names, except for attribContainer which is for tag attributes. Also, without post-processing, no information is lost

What is required is a method to strip the markup and extract the text data as strings into a nested list expression preserving the (tree) structure. Thus for the above portion, the desired output is:

 {"Subjective",
    {"XX yr old man here for fu on med problems",
     "feels absolutely positively wonderful"},
  "Active problems",
    {"htn",
         {"started on hctz in 8/10, in early 11 given lasix as his gfr was low",
          "limits salt caffeine, home readings140/80  or less when he runs",
          "not been exercising now"}}}

I made the entire HTML document available online:

parsed = postProcess@
   parseText[
    Import["http://statigrafix.com/projects/pace/datasets/20130527/\
NOTES.001-01.0.WORKFLOWY.20130527.html", "TEXT"]];

From manually analyzing the example () and following the tree structure, it's easy to extract at the top level, eg:

Column@Cases[parsed[[1, 5, 5]], 
  liContainer[attribContainer[_], txt_] :> txt]

(*
Subjective
Active problems
General health
Past medical and surgical history
DA
Medications
8/08  Family History
8/08  Personal and social history
Habits
Exam
Labs
Clinical Reminder Activity
*)

I would appreciate help with the method to automate extraction at all levels.

$\endgroup$
3
  • 2
    $\begingroup$ What is wrong with the following? data = Import["http://statigrafix.com/projects/pace/datasets/20130527/NOTES.001-01.0.WORKFLOWY.20130527.html", {"HTML", "XMLObject"}]; then Cases[data, XMLElement["body", ___], \[Infinity]] //. XMLElement[_, _, content_] :> content $\endgroup$
    – Szabolcs
    May 27, 2013 at 17:50
  • $\begingroup$ I did not know about HTML --> XMLObject conversion, Thanks. Your pattern does exctract the data, however, it doesn't seem to be preserving the tree structure. Eg, check the Dimensions . We're looking to preserve the nesting. $\endgroup$ May 27, 2013 at 18:02
  • 1
    $\begingroup$ I'm not sure what you mean. It does preserve the tree structure precisely as it is in the HTML. The code needs to be improved a bit to get rid of extra nesting, but the structure is precisely preserved. $\endgroup$
    – Szabolcs
    May 27, 2013 at 19:02

1 Answer 1

4
$\begingroup$

The benefit of using a symbolic tree representation with inert heads as in Leonid's parser is that you can then decide how to represent the data. And that is indeed what you should do, instead of extracting the elements using Cases. Here's an example using your parsed output above:

Block[{ulContainer, liContainer},
    ulContainer[_, l__] := {l};
    liContainer[_, s_String] := s;
    parsedExpr /. Null -> Sequence[]
]

where parsedExpr is as in the question. This gives you:

{   
    "Subjective", 
    {
        "XX yr old man here for fu on med problems", 
        "feels absolutely positively wonderful"
    }, 
    "Active problems", 
    {
        "htn", 
        {
            "started on hctz in 8/10,in early 11 given lasix as his gfr was low", 
            "limits salt caffeine,home readings140/80 or less when he runs", 
            "not been exercising now"
        }
    }
}

You can see that this preserves the tree structure.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.