# Extracting parts of nested lists of rules using a syntax similar to JsonPath or Xpath

Given an example data set:

data = {"region" -> "AA",
"systems" -> {{"name" -> 1,
"sub" -> {{"name" -> "sub11"}, {"name" -> "sub12"}}},
{"name" -> 2,
"sub" -> {{"name" -> "sub21"}, {"name" -> "sub22"}}}}};


I would like to extract the various systems and subsystems attributes. I would prefer a sysntax similar to JsonPath ot XPath.

So for example, I would like to be able to write something similar to:

data."region" (* AA *)
data."systems"[2]."name" (* 2 *)
data."systems"[2]."sub"[2]."name" (* sub22 *)


Filtering is a nice bonus

 data."systems"[#."name" == 1 &]."sub"."name" (* {"sub11", "sub12"} *)


How could I make this or a similar syntax work?

Using ReplaceRepeated almost works (just the substitutions have to be written in reverse ie "name" /. "systems"[[2]] instead of the more logical "systems"[[2]] -> "name").

{"region", "name" /. "systems"[[2]], "sub"[[2]] /. "systems"[[2]]} //. data
(* ==> {"AA", 2, {"name" -> "sub22"}} *)


But this fails, as "name" appears on two levels

{"region", "name" /. "systems"[[2]],
"name" /. "sub"[[2]] /. "systems"[[2]]} //. data

(* ==> {"AA", 2, 2}  but I would like {"AA", 2, sub22}   *)

-
I think that your option list does not qualify as a valid option list, at least in the sense you wish it to be. Options are fundamentally named arguments (and not positional). So, the name, or sequence of names at each level, should uniquely define a value. In your case, your first and second sub-system options are having the same set of names, so it is impossible to distinguish between them without taking their positions into account. If you are asking about generalized options which are also positional, you should clearly say so, but be warned that this is (IMO) a shaky ground. –  Leonid Shifrin Jul 16 '12 at 9:23
I understand. I just think that in your case, options abstraction is wrong, and trying to shoehorn it into the one where you could use options-related functionality does not strike me as a good idea. Rules and options buy you position-independence, but only of your data can be accessed in a position-independent way. I would either reformulate the problem (e.g. split your data set into several each of which represents a valid option set), or use different data structures, which can distinguish elements based on their position in them. –  Leonid Shifrin Jul 16 '12 at 9:32
@Szabolcs Thanks, I indeed did not. I can't seem to be able to read the draft from my WordPress blog account though, although I see the draft listed in the "Posts" tab. Is it the correct behavior (I assume that only admin can see read drafts)? –  Leonid Shifrin Jul 16 '12 at 9:42
@LeonidShifrin Hmm perhaps I should just make this into a relational table and then use SQL. But I was hoping to avoid that... The elements I would like to access are well defined. I just find OptionValue[data, "systems"[[2]] -> "sub"[[2]] -> "name"] easier to read than OptionValue[ OptionValue[OptionValue[data, "systems"][[2]], "sub"][[2]], "name"]. I guess I would like some sort of Xpath approximation where it is also possible to get nodes either by position or name. –  Ajasja Jul 16 '12 at 9:51
Aha, Including Xpath in my search yields something interesting mathematica.stackexchange.com/questions/3916/… –  Ajasja Jul 16 '12 at 9:58

Would something like this be suitable?

sys_~s~sub_ := sub /. sys
sys_~s~sub_[n_] := (sub /. sys)[[n]]

data~s~"region"
data~s~"systems"[2]~s~"name"
data~s~"systems"[2]~s~"sub"[2]
data~s~"systems"[2]~s~"sub"[2]~s~"name"


"AA"

2

{"name" -> "sub22"}

"sub22"

-
A great idea, thanks! (I replaced ~s~ with \[CircleMinus], but that is just a matter of preference) –  Ajasja Jul 16 '12 at 13:51
It would also be nice to be able to specify All and Span with tree-structured key-value pair data (eg XML); I asked a similar question: mathematica.stackexchange.com/questions/3325/… –  alancalvitti Jul 16 '12 at 17:27
@Ajasja, good idea. I forget that Mathematica has all those infix operators. –  Simon Woods Jul 16 '12 at 21:01
@alancalvitti, I'm not sure what All and Span mean in the context of data structures and XML, it's not really my thing. I assume you are aware of Mathematica's Symbolic XML functionality? –  Simon Woods Jul 16 '12 at 21:06
@SimonWoods, I use symbolic XML; attributes are stored as key->value pairs. All is easy to explain, it would return all branches from a given node in the data tree. Span would only work on hybrid tree/relational data structures, eg where data at the lower levels is relational matrix, but to access and aggregate it XPath type indexing could be nice syntactic sugar. Routinely in data analysis, I use such hybrid data structures. –  alancalvitti Jul 16 '12 at 21:15

With a small modification to @SimonWoods answer it's possible to filter the nodes as well:

Clear[CircleMinus]
sys_\[CircleMinus]sub_ := sub /. sys
sys_\[CircleMinus]sub_[n_] := (sub /. sys)[[n]]
sys_\[CircleMinus]sub_[f_Function] := Module[{s},
s = Select[(sub /. sys), f];
(*remove redundant {}*)
s /. {x_List} :> x];


Now, one can do things like select only system nodes that have a particular value of the "name" attribute:

data\[CircleMinus]"systems"[#\[CircleMinus]"name" >=1 &]\[CircleMinus]"sub"[2]\[CircleMinus]"name"
data\[CircleMinus]"systems"[#\[CircleMinus]"name" ==1 &]\[CircleMinus]"sub"[1]\[CircleMinus]"name"
data\[CircleMinus]"systems"[#\[CircleMinus]"name" ==1 &]\[CircleMinus]"sub"\[CircleMinus]"name"


(This really does look better in the notebook)

Test that the previous syntax still works:

data\[CircleMinus]"region"
data\[CircleMinus]"systems"[2]\[CircleMinus]"name"
data\[CircleMinus]"systems"[2]\[CircleMinus]"sub"[2]


Note that this is still note well tested. For example I'm not sure why I need If [Length@s == 1 && Head[s[[1]] ] === List, s[[1]], s]. But I got some redundant {{}} and this is hacky way to remove them.

With some more effort it's also possible to override Dot

Attributes[withDotPath] = HoldFirst;
withDotPath[code_] := InternalInheritedBlock[{Dot}, Unprotect[Dot];
Dot[sys_, sub_] := sub /. sys;
Dot[arg1_, arg2_, arg3_] := Dot[Dot[arg1, arg2], arg3];
Dot[arg1_, arg2_, args__] := Dot[Dot[arg1, arg2], args];
Dot[sys_, sub_[n_]] := (sub /. sys)[[n]];
Dot[sys_, sub_[f_Function]] := Module[{s},
s = Select[(sub /. sys), f]; (*remove redundant {}*)
s /. {x_List} :> x];
code
];


Then one can use the following beautiful syntax:

withDotPath[Column@{
data."region",
data."systems"[2]."name",
data."systems"[2]."sub"[2],
data."systems"[#."name" >= 1 &]."sub"[2]."name",
data."systems"[#."name" == 1 &]."sub"[1]."name",
data."systems"[#."name" == 1 &]."sub"."name"
}]


Any comments or further improvements are most welcome.

-
Nice work! I think the {{}} is because Select always returns a list of results, even if there is only one of them. A neater (IMO) way to strip the excess brackets is s/.{x_List}:>x. –  Simon Woods Jul 17 '12 at 11:04
@SimonWoods Thanks! I included your suggestion, but kept the code a bit more verbose. The shortest way would be. Dot[sys_, sub_[f_Function]] := Select[(sub /. sys), f] /. {x_List} :> x;` –  Ajasja Jul 17 '12 at 11:19