Extracting text from nested HTML lists <ul> <li>

Question

We wish to analyze the tree structure of HTML documents that consist of nested lists, ie, using the HTML tags <ul> and <li>. The documents are generated by Workflowy, a note taking web app that can manage arbitrarily deep nesting (and dynamically re-arrange them by zooming and searching).

For example, here's a portion of one document:

<ul class="ul1">
  <li class="li2">Subjective</li>
  <ul class="ul1">
    <li class="li2">XX yr old man here for fu on med problems</li>
    <li class="li2">feels absolutely positively wonderful</li>
  </ul>
  <li class="li2">Active problems</li>
  <ul class="ul1">
    <li class="li2">htn</li>
    <ul class="ul1">
      <li class="li2">started on hctz in 8/10, in early 11 given lasix as his gfr was low</li>
      <li class="li2">limits salt caffeine, home readings140/80  or less when he runs</li>
      <li class="li2">not been exercising now</li>
    </ul>

After applying the main parse function in Leonid Shifrin's HTML parser (Thanks for that), the above portion is transcoded to:

ulContainer[attribContainer[ class="ul1"],
  ,liContainer[attribContainer[ class="li2"],Subjective],
  ,ulContainer[attribContainer[ class="ul1"],
    ,liContainer[attribContainer[ class="li2"],XX yr old man here for fu on med problems],
    ,liContainer[attribContainer[ class="li2"],feels absolutely positively wonderful],
  ],
  ,liContainer[attribContainer[ class="li2"],Active problems],
  ,ulContainer[attribContainer[ class="ul1"],
    ,liContainer[attribContainer[ class="li2"],htn],
    ,ulContainer[attribContainer[ class="ul1"],
      ,liContainer[attribContainer[ class="li2"],started on hctz in 8/10, in early 11 given lasix as his gfr was low],
      ,liContainer[attribContainer[ class="li2"],limits salt caffeine, home readings140/80  or less when he runs],
      ,liContainer[attribContainer[ class="li2"],not been exercising now],
    ],

Shifrin writes:

What it gives you is a symbolic tree representation of the parsed html document, with different tags transformed into ...Container[content], for example divContainer[aContainer["Some text"]]. The container names mirror the html tag names, except for attribContainer which is for tag attributes. Also, without post-processing, no information is lost

What is required is a method to strip the markup and extract the text data as strings into a nested list expression preserving the (tree) structure. Thus for the above portion, the desired output is:

 {"Subjective",
    {"XX yr old man here for fu on med problems",
     "feels absolutely positively wonderful"},
  "Active problems",
    {"htn",
         {"started on hctz in 8/10, in early 11 given lasix as his gfr was low",
          "limits salt caffeine, home readings140/80  or less when he runs",
          "not been exercising now"}}}

I made the entire HTML document available online:

parsed = postProcess@
   parseText[
    Import["http://statigrafix.com/projects/pace/datasets/20130527/\
NOTES.001-01.0.WORKFLOWY.20130527.html", "TEXT"]];

From manually analyzing the example () and following the tree structure, it's easy to extract at the top level, eg:

Column@Cases[parsed[[1, 5, 5]], 
  liContainer[attribContainer[_], txt_] :> txt]

(*
Subjective
Active problems
General health
Past medical and surgical history
DA
Medications
8/08  Family History
8/08  Personal and social history
Habits
Exam
Labs
Clinical Reminder Activity
*)

I would appreciate help with the method to automate extraction at all levels.

Answer 1

The benefit of using a symbolic tree representation with inert heads as in Leonid's parser is that you can then decide how to represent the data. And that is indeed what you should do, instead of extracting the elements using Cases. Here's an example using your parsed output above:

Block[{ulContainer, liContainer},
    ulContainer[_, l__] := {l};
    liContainer[_, s_String] := s;
    parsedExpr /. Null -> Sequence[]
]

where parsedExpr is as in the question. This gives you:

{   
    "Subjective", 
    {
        "XX yr old man here for fu on med problems", 
        "feels absolutely positively wonderful"
    }, 
    "Active problems", 
    {
        "htn", 
        {
            "started on hctz in 8/10,in early 11 given lasix as his gfr was low", 
            "limits salt caffeine,home readings140/80 or less when he runs", 
            "not been exercising now"
        }
    }
}

You can see that this preserves the tree structure.