Parsing and Extracting XML Content

Question

I've run into a little question while trying to parse XML. This seems like a rather simple task, but for the life of me I cannot figure it out. I pull up the XML of the website using the following code:

url = "https://investorshub.advfn.com/Twilio-TWLO-31145/?NextStart=0";
getXml[anyUrl_] := (
   current = url;
   page = URLFetch[url];
   If[StringLength@page == 0, getXml@current, 
    Return@ImportString[page, "XMLObject"]]);
largeXml = getXml[url];

This returns to me a very long XML Tree. But what I am looking at specifically is this piece:

  XMLElement[
   "a", {"shape" -> "rect", "id" -> "ctl00_CP1_gv_ctl03_hlSub", 
    "href" -> 
     "/boards/read_msg.aspx?message_id=131676511"}, {"wow this thing \
trades all over the place."}]

The overarching goal is to extract the part of the XML that says "wow this thing trades all over the place." I've done that with the following code:

firstScreen = Cases[largeXml,
  XMLElement[
   "a", {"shape" -> "rect", "id" -> "ctl00_CP1_gv_ctl03_hlSub", 
    "href" -> "/boards/read_msg.aspx?message_id=131676511"}, {___}], 
  Infinity]
secondScreen = Cases[#,
    XMLElement[
      "a", {"shape" -> "rect", "id" -> "ctl00_CP1_gv_ctl03_hlSub", 
       "href" -> 
        "/boards/read_msg.aspx?message_id=131676511"}, {comment_}] -> 
     comment, 1] &@firstScreen 

"wow this thing trades all over the place" is a subject header on the website that leads into a comment page. There are various subject headers on the webpage all with the same "href" except for the id #. In other words, all of the comments have the XML pattern:

XMLElement[
          "a", {"shape" -> "rect", "id" -> "ctl00_CP1_gv_ctl03_hlSub", 
           "href" -> 
            "/boards/read_msg.aspx?message_id=131676511"}, {comment_}] 

The thing that differentiates them is the id # ... "message_id = something different for each subject header". Is there a way I can make it so that I can pull up all the subject headers at once? I've thought of this:

firstScreen = Cases[largeXml,
  XMLElement[
   "a", {"shape" -> "rect", "id" -> "ctl00_CP1_gv_ctl03_hlSub", 
    "href" -> "/boards/read_msg.aspx?message_id=___"}, {___}], 
  Infinity]

But that does not return anything. Also, I thought of RegEx but have also heard that RegEx cannot be used to parse HTML or XML. Any help would be greatly appreciated, thank you!

Answer 1

xml = { XMLElement[    "a", {
     "shape" -> "rect", 
     "id" -> "ctl00_CP1_gv_ctl03_hlSub", 
     "href" -> "/boards/read_msg.aspx?message_id=131676511"
   }, 
 {comment_}]
};

Cases[xml
  , XMLElement[_, {___, "href" -> (
        s_?(StringStartsQ["/boards/read_msg.aspx?message_id="])
      ), ___}, _
    ] :> s
  , ∞
]

{"/boards/read_msg.aspx?message_id=131676511"}

or:

Cases[xml, XMLElement[_,  KeyValuePattern[ 
  "href" -> (s_?(StringStartsQ["/boards/read_msg.aspx?message_id="]))
 ], _] :>   s, ∞
]