10
$\begingroup$

I'm interested in working with XML more conveniently and efficiently than I currently can. The standard XMLDocument/XMLElement structure is incredibly inefficient.

My thought was to move to something like:

XMLGraph[nodeGraph_Graph, nodeData_Association]

where the nodeGraph is the tree structure of the XML and the nodeData is an Association of data referenced into by the nodeGraph.

I could then define standard access/mutation mechanisms that'd be efficient and also allow me to do use BreadthFirstScan and things for parsing, as well as immediately extract node data from the Association.

Can anyone think of a better way? (Or think of a clean way to implement this?)

$\endgroup$
9
$\begingroup$

I'm going to focus on the convenience part of the question. jsoupLink can work well for a lot of situations. Let's say, for example, that we want to parse RSS. RSS is an XML format, so we can do it like this:

<< jSoupLink`
root = Import["http://packagedata.net/index.php/feed", "HTMLDOM"]
items = root["Select", "item"];

Mathematica graphics

What this shows is that using jSoupLink, we can often retrieve information with a much cleaner syntax than if we were using Mathematica's symbolic XML.

jsoupLink can also modify elements. Here's an example of how we can build an SVG, which is also an XML format, iteratively:

root = ImportString[
   "<svg  xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"300\" height=\"300\"></svg>",
   "HTMLDOM"
   ];
svg = First[root["Select", "svg"]];

Do[
  svg[
   "Append",
   "<circle cx=\"50\" cy=\"50\" r=\"10\" fill=\"blue\"/>"],
  {10}
  ];

Export["~/Desktop/test.svg", svg, "HTMLDOM"]

If we look at this SVG file in a browser, we see the following (posted here as a PNG):

Example

All the disks are on top of each other and of the same color. We can now also use jsoupLink to shift them around and change their color:

circles = svg["Select", "circle"];
Do[
  Part[circles, i]["Attribute", "cx", ToString@RandomReal[100]];
  Part[circles, i]["Attribute", "cy", ToString@RandomReal[100]];
  Part[circles, i]["Attribute", "fill", Image`Utilities`toHEXcolor@ColorData[97, i]];
  ,
  {i, Length[circles]}
  ];

ExportString[svg, "HTMLDOM"]

(* Out:

"<svg xmlns=\"http://www.w3.org/2000/svg\" \
xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"300\" \
height=\"300\">
 <circle cx=\"39.6123\" cy=\"98.7366\" r=\"10\" fill=\"#5e81b5\" />
 <circle cx=\"42.4556\" cy=\"71.0983\" r=\"10\" fill=\"#e19c24\" />
 <circle cx=\"26.9683\" cy=\"9.98416\" r=\"10\" fill=\"#8fb032\" />
 <circle cx=\"36.7453\" cy=\"66.7649\" r=\"10\" fill=\"#eb6235\" />
 <circle cx=\"94.7156\" cy=\"21.069\" r=\"10\" fill=\"#8778b3\" />
 <circle cx=\"77.1975\" cy=\"61.0884\" r=\"10\" fill=\"#c56e1a\" />
 <circle cx=\"21.5307\" cy=\"24.9411\" r=\"10\" fill=\"#5d9ec7\" />
 <circle cx=\"13.0563\" cy=\"62.826\" r=\"10\" fill=\"#ffbf00\" />
 <circle cx=\"24.5768\" cy=\"57.8981\" r=\"10\" fill=\"#a5609d\" />
 <circle cx=\"24.2052\" cy=\"21.9666\" r=\"10\" fill=\"#929600\" />
</svg>"

*)

Now it looks like this:

Example 2

$\endgroup$
  • $\begingroup$ That is very convenient. How easy is it to get out properties, like scan for a certain XML structure pattern? $\endgroup$ – b3m2a1 Oct 20 '18 at 16:15
  • $\begingroup$ @b3m2a1 That's what the node["Select", "css selector"] function is for. CSS selectors excel at describing how elements relate to each other hierarchically. $\endgroup$ – C. E. Oct 20 '18 at 18:39
  • $\begingroup$ Ahh. Very nice. One fun thing, I see you have some property assignment syntax like circles[[i]]["Attribute", "cx", ToString@RandomReal[100]], have you thought about using the new MutationHandler stuff to make that more standard? e.g. you could do node["name"]=val and it'd unwrap to node["Attribute", "name", ToString@val]? Maybe also supporting AssociateTo. In any case, as far as convenience goes I think that'd be about as convenient as it comes, right? The MutationHandler would be necessary for things like when node has been assigned to a variable. $\endgroup$ – b3m2a1 Oct 20 '18 at 18:43
  • 1
    $\begingroup$ @b3m2a1 I just heard about MutationHandler yesterday (from you) and I don't know what it is yet but that sounds like exactly what I would like to do actually. I'll keep that in mind. $\endgroup$ – C. E. Oct 20 '18 at 18:52
4
$\begingroup$

C.E.'s answer is probably the best, relying as it does on a well-constructed Java library with serious people behind it, but I thought it'd be nice to have a pure Mathematica solution as well, so I sunk some time into building an XMLGraph object which I put here.

The basic idea is to store the two pieces of data I talked about before, a Graph that holds all of the structural relationships, and an Association encoding all of the data as a flat data structure.

Each node then gets a name, e.g. "head:ee1d9544" which goes into the Graph and an entry in the Association like:

<|
  "Parent" -> "html:80da223b",
  "Type" -> "head",
  "Name" -> "head:ee1d9544",
  "Meta" -> <||>,
  "Children" -> {}
|>

This way it's easy to mutate the object by changing entries in the flat association and it's easy to query the Graph to get out structure.

I then used this to implement conversion from XMLElement and CSS selectors, so here's a little rundown of what we can do (although development work isn't finished):

Get["https://github.com/b3m2a1/mathematica-tools/raw/master/XMLGraph.m"];

testXML = 
  Import["https://developer.github.com/apps/building-oauth-apps/understanding-scopes-for-oauth-apps/#available-scopes",
   {"HTML", "XMLObject"}
   ];

g = XMLGraph[testXML]

enter image description here

Then we can query, say, all the code elements that follow a td element (this was what originally spurred me to make this). The Mathematica code for this query in the standard way looks like:

t1 = Cases[testXML, 
  XMLElement["td", _, code_] :> 
   FirstCase[code, XMLElement["code", _, {s_, ___}] :> s, 
    Nothing, \[Infinity]], \[Infinity]];

The graph query looks like:

t2 = First /@ Values@
  g@"Children"[g@"Select"["td code"]];

DeleteDuplicates@Sort@t1 == DeleteDuplicates@Sort@t2

True

Performance-wise we're a bit slower:

t1 = Cases[testXML, 
    XMLElement["td", _, code_] :> 
     FirstCase[code, XMLElement["code", _, {s_, ___}] :> s, 
      Nothing, \[Infinity]], \[Infinity]]; // RepeatedTiming

{0.00045, Null}

t2 = First /@ Values@
     g@"Children"[g@"Select"["td code"]]; // RepeatedTiming

{0.0091, Null}

But other things like property look-ups are so much faster like this. Also we can do queries over the Association instead of the graph (when possible) which is competitive with Cases:

Cases[testXML, 
   XMLElement["code", _, _], \[Infinity]]; // RepeatedTiming

{0.00015, Null}

nodes = g@"Select"["code"]; // RepeatedTiming

{0.00077, Null}

And once we have this set of nodes it's easy and fast to get all sorts of information out:

g@"Children"[nodes]; // RepeatedTiming

{0.00012, Null}

g@"Parent"[nodes]; // RepeatedTiming

{0.000082, Null}

g@"Attribute"[nodes, "colspan"]; // RepeatedTiming

{0.000084, Null}

This is rather more challenging to do well with XMLElement.

We can also do easy modifications like this:

newG = g@"ModifyNodes"[
    Thread[nodes -> <|"colspan" -> 2|>]
    ];

g@"Attribute"[nodes, "colspan"] // Take[#, 5] &

<|"td:ae4c93a0" -> "1", "td:c3fbb1ce" -> "1", "td:77adbfa1" -> "1", 
 "td:3efd9c29" -> "1", "td:69452f14" -> "1"|>

newG@"Attribute"[nodes, "colspan"] // Take[#, 5] &

<|"td:ae4c93a0" -> 2, "td:c3fbb1ce" -> 2, "td:77adbfa1" -> 2, 
 "td:3efd9c29" -> 2, "td:69452f14" -> 2|>

Or we can change some types and dump to XML:

mods = {newG["Root"] -> <|"Type" -> "xml"|>};
newG@"ModifyNodes"[mods]@"XML" // Shallow

XMLElement["xml", <|"version" -> 
   "-//W3C//DTD HTML 4.01 Transitional//EN", "lang" -> "en", 
  "prefix" -> 
   "og: http://ogp.me/ns#", {"http://www.w3.org/2000/xmlns/", 
    "xmlns"} -> "http://www.w3.org/1999/xhtml"|>, {XMLElement[<<3>>], 
  XMLElement[<<3>>]}]

The full list of supported methods is somewhat in flux, but it'll be exposed eventually and every method has a Symbol backing it (e.g. XMLGraph`Package`SelectNodes for "Select") so that's always possible to look at.

As a final usage note, with the structure as it's set-up the storage can always be handled by as HashTable which would allow this XMLGraph to be a truly mutable object in the way the object-oriented interface suggests it could be.

Hopefully this gives a sense of how this may be done and why this type of structure is useful. And of course, this is a preliminary package, so performance enhancements are always possible.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.