Efficient structure for XML manipulation

Question

I'm interested in working with XML more conveniently and efficiently than I currently can. The standard XMLDocument/XMLElement structure is incredibly inefficient.

My thought was to move to something like:

XMLGraph[nodeGraph_Graph, nodeData_Association]

where the nodeGraph is the tree structure of the XML and the nodeData is an Association of data referenced into by the nodeGraph.

I could then define standard access/mutation mechanisms that'd be efficient and also allow me to do use BreadthFirstScan and things for parsing, as well as immediately extract node data from the Association.

Can anyone think of a better way? (Or think of a clean way to implement this?)

Answer 1

I'm going to focus on the convenience part of the question. jsoupLink can work well for a lot of situations. Let's say, for example, that we want to parse RSS. RSS is an XML format, so we can do it like this:

<< jSoupLink`
root = Import["http://packagedata.net/index.php/feed", "HTMLDOM"]
items = root["Select", "item"];

What this shows is that using jSoupLink, we can often retrieve information with a much cleaner syntax than if we were using Mathematica's symbolic XML.

jsoupLink can also modify elements. Here's an example of how we can build an SVG, which is also an XML format, iteratively:

root = ImportString[
   "<svg  xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"300\" height=\"300\"></svg>",
   "HTMLDOM"
   ];
svg = First[root["Select", "svg"]];

Do[
  svg[
   "Append",
   "<circle cx=\"50\" cy=\"50\" r=\"10\" fill=\"blue\"/>"],
  {10}
  ];

Export["~/Desktop/test.svg", svg, "HTMLDOM"]

If we look at this SVG file in a browser, we see the following (posted here as a PNG):

All the disks are on top of each other and of the same color. We can now also use jsoupLink to shift them around and change their color:

circles = svg["Select", "circle"];
Do[
  Part[circles, i]["Attribute", "cx", ToString@RandomReal[100]];
  Part[circles, i]["Attribute", "cy", ToString@RandomReal[100]];
  Part[circles, i]["Attribute", "fill", Image`Utilities`toHEXcolor@ColorData[97, i]];
  ,
  {i, Length[circles]}
  ];

ExportString[svg, "HTMLDOM"]

(* Out:

"<svg xmlns=\"http://www.w3.org/2000/svg\" \
xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"300\" \
height=\"300\">
 <circle cx=\"39.6123\" cy=\"98.7366\" r=\"10\" fill=\"#5e81b5\" />
 <circle cx=\"42.4556\" cy=\"71.0983\" r=\"10\" fill=\"#e19c24\" />
 <circle cx=\"26.9683\" cy=\"9.98416\" r=\"10\" fill=\"#8fb032\" />
 <circle cx=\"36.7453\" cy=\"66.7649\" r=\"10\" fill=\"#eb6235\" />
 <circle cx=\"94.7156\" cy=\"21.069\" r=\"10\" fill=\"#8778b3\" />
 <circle cx=\"77.1975\" cy=\"61.0884\" r=\"10\" fill=\"#c56e1a\" />
 <circle cx=\"21.5307\" cy=\"24.9411\" r=\"10\" fill=\"#5d9ec7\" />
 <circle cx=\"13.0563\" cy=\"62.826\" r=\"10\" fill=\"#ffbf00\" />
 <circle cx=\"24.5768\" cy=\"57.8981\" r=\"10\" fill=\"#a5609d\" />
 <circle cx=\"24.2052\" cy=\"21.9666\" r=\"10\" fill=\"#929600\" />
</svg>"

*)

Now it looks like this:

Answer 2

C.E.'s answer is probably the best, relying as it does on a well-constructed Java library with serious people behind it, but I thought it'd be nice to have a pure Mathematica solution as well, so I sunk some time into building an XMLGraph object which I put here.

The basic idea is to store the two pieces of data I talked about before, a Graph that holds all of the structural relationships, and an Association encoding all of the data as a flat data structure.

Each node then gets a name, e.g. "head:ee1d9544" which goes into the Graph and an entry in the Association like:

<|
  "Parent" -> "html:80da223b",
  "Type" -> "head",
  "Name" -> "head:ee1d9544",
  "Meta" -> <||>,
  "Children" -> {}
|>

This way it's easy to mutate the object by changing entries in the flat association and it's easy to query the Graph to get out structure.

I then used this to implement conversion from XMLElement and CSS selectors, so here's a little rundown of what we can do (although development work isn't finished):

Get["https://github.com/b3m2a1/mathematica-tools/raw/master/XMLGraph.m"];

testXML = 
  Import["https://developer.github.com/apps/building-oauth-apps/understanding-scopes-for-oauth-apps/#available-scopes",
   {"HTML", "XMLObject"}
   ];

g = XMLGraph[testXML]

Then we can query, say, all the code elements that follow a td element (this was what originally spurred me to make this). The Mathematica code for this query in the standard way looks like:

t1 = Cases[testXML, 
  XMLElement["td", _, code_] :> 
   FirstCase[code, XMLElement["code", _, {s_, ___}] :> s, 
    Nothing, \[Infinity]], \[Infinity]];

The graph query looks like:

t2 = First /@ Values@
  g@"Children"[g@"Select"["td code"]];

DeleteDuplicates@Sort@t1 == DeleteDuplicates@Sort@t2

True

Performance-wise we're a bit slower:

t1 = Cases[testXML, 
    XMLElement["td", _, code_] :> 
     FirstCase[code, XMLElement["code", _, {s_, ___}] :> s, 
      Nothing, \[Infinity]], \[Infinity]]; // RepeatedTiming

{0.00045, Null}

t2 = First /@ Values@
     g@"Children"[g@"Select"["td code"]]; // RepeatedTiming

{0.0091, Null}

But other things like property look-ups are so much faster like this. Also we can do queries over the Association instead of the graph (when possible) which is competitive with Cases:

Cases[testXML, 
   XMLElement["code", _, _], \[Infinity]]; // RepeatedTiming

{0.00015, Null}

nodes = g@"Select"["code"]; // RepeatedTiming

{0.00077, Null}

And once we have this set of nodes it's easy and fast to get all sorts of information out:

g@"Children"[nodes]; // RepeatedTiming

{0.00012, Null}

g@"Parent"[nodes]; // RepeatedTiming

{0.000082, Null}

g@"Attribute"[nodes, "colspan"]; // RepeatedTiming

{0.000084, Null}

This is rather more challenging to do well with XMLElement.

We can also do easy modifications like this:

newG = g@"ModifyNodes"[
    Thread[nodes -> <|"colspan" -> 2|>]
    ];

g@"Attribute"[nodes, "colspan"] // Take[#, 5] &

<|"td:ae4c93a0" -> "1", "td:c3fbb1ce" -> "1", "td:77adbfa1" -> "1", 
 "td:3efd9c29" -> "1", "td:69452f14" -> "1"|>

newG@"Attribute"[nodes, "colspan"] // Take[#, 5] &

<|"td:ae4c93a0" -> 2, "td:c3fbb1ce" -> 2, "td:77adbfa1" -> 2, 
 "td:3efd9c29" -> 2, "td:69452f14" -> 2|>

Or we can change some types and dump to XML:

mods = {newG["Root"] -> <|"Type" -> "xml"|>};
newG@"ModifyNodes"[mods]@"XML" // Shallow

XMLElement["xml", <|"version" -> 
   "-//W3C//DTD HTML 4.01 Transitional//EN", "lang" -> "en", 
  "prefix" -> 
   "og: http://ogp.me/ns#", {"http://www.w3.org/2000/xmlns/", 
    "xmlns"} -> "http://www.w3.org/1999/xhtml"|>, {XMLElement[<<3>>], 
  XMLElement[<<3>>]}]

The full list of supported methods is somewhat in flux, but it'll be exposed eventually and every method has a Symbol backing it (e.g. XMLGraph`Package`SelectNodes for "Select") so that's always possible to look at.

As a final usage note, with the structure as it's set-up the storage can always be handled by as HashTable which would allow this XMLGraph to be a truly mutable object in the way the object-oriented interface suggests it could be.

Hopefully this gives a sense of how this may be done and why this type of structure is useful. And of course, this is a preliminary package, so performance enhancements are always possible.