The summary box for Dataset
objects shows the number of levels and elements at the bottom.
How can we get these values programmatically?
For example, "4 levels, 350 elements" in the Planets dataset:
Mathematica Stack Exchange is a question and answer site for users of Wolfram Mathematica. It only takes a minute to sign up.
Sign up to join this communityIf we are not afraid of using undocumented functions, we can get the counts the same way that the summary box gets them:
info @ Dataset[data_, type_, _] :=
{TypeSystem`TypeDepth[type], TypeSystem`AtomCount[type, data]}
info @ ExampleData[{"Dataset", "Planets"}]
(* {4, 350} *)
Or, as per @Szabolcs' suggestion:
info2 @ ds_Dataset :=
{TypeSystem`TypeDepth[#], TypeSystem`AtomCount[##]}& @@
Through[{Dataset`GetType, Dataset`GetData} @ ds]
info2 @ ExampleData[{"Dataset", "Planets"}]
(* {4, 350} *)
TypeSystem
and couldn't find them. Grrrr +1
$\endgroup$
Dataset`GetType[]
(still undocumented, of course).
$\endgroup$
This is not quite the same thing, but strongly related and very useful to know:
It appears that the undocumented option AllowedHeads
, which appears in several functions in v10, can help here.
ds = ExampleData[{"Dataset", "Planets"}];
To get the depth,
ArrayDepth[
Normal[ds],
AllowedHeads -> {Association, List}
]
This also works:
ArrayDepth[
Normal[ds],
AllowedHeads -> All
]
To get the element count from this generalized array,
Times @@ Dimensions[
Normal[ds],
AllowedHeads -> All
]
These counts refer only to the rectangular part of the generalized array made of lists and associations. They won't give the counts 4 and 350. For that we'd need to descend deeper.
Out of curiosity, this is the complete list of System`
functions having this option:
ArrayDepth, ConjugateTranspose, Depth, Dimensions, Transpose
Transpose
and ConjugateTranspose
are useful for taking the transpose of mixed List
-Association
structures.
Depth
would theoretically be good for finding the number of levels in the dataset, but I don't know how to use this option with it. It seems to take only the values True
and False.
ArrayDepth
is same as Dimensions /* Length
. But Dimensions
itself is more informative for many data processing tasks.
$\endgroup$
Oct 7, 2014 at 18:49
There's at least the brute force approach
getFromDs[what_String][ds_Dataset] :=
FirstCase[ToBoxes@ds,
RowBox[{___, i_, Shortest@___, what, ___}] :>
With[{res = ToExpression@i}, res /; IntegerQ[res]], $Failed,
Infinity]
getLevels = getFromDs["levels"]; getElements = getFromDs["elements"];
Through@{getLevels, getElements}@ExampleData[{"Dataset", "Planets"}]
(* {4, 350} *)
This is an attempt at a disgusting function for the depth: the maximum number of lists or associations you can go through from the top down before bouncing in something else
We setup some tests
test[dep] ^= Inactivate@{
dep[{1}] -> 1,
dep[4] -> 0,
dep[h[4, g[8]]] -> 0,
dep[<|"a" -> 3|>] -> 1,
dep[<|2|>] -> 0,
dep[<|"a" -> 2|>] -> 1,
dep[{<|"a" -> 2|>, d}] -> 2,
dep[<|{2}|>] -> 0,
dep[<|"a" -> {2}, "b" -> 8|>] -> 2,
dep[<|"a" :> {<|"c" -> h[2]|>}|>] -> 3,
dep[<|"x" -> {1}, "x" -> 3|>] -> 1,
dep[<|"x" :> <|"x" -> {1}, "x" -> 3|>|>] -> 2
};
runTest[sym_] :=
Inactive[TestReport][
test[sym] /. (x_ -> y_) :> Inactive[VerificationTest][x, y]] //
Activate
The code
dep[ds_Dataset] := dep[Normal@ds];
dep[stuff_] := idep[0, stuff];
SetAttributes[idep, HoldAllComplete];
idep[i_, l_List |
l_Association?(Function[i, AssociationQ@Unevaluated@i,
HoldAllComplete])] :=
1 + Max[Function[x, idep[i, x], HoldAllComplete] /@
Unevaluated@l /. {a_Association :> Values@a}];
idep[i_, a_Association /;
MatchQ[Unevaluated@a,
HoldPattern[
Association[(_Rule | _RuleDelayed) ... | {(_Rule | \
_RuleDelayed) ...}]]]] :=
Module[{v = Block[{Association = HoldComplete}, a]},
With[{rhs = v[[All, 2]]},
1 + Max[Function[x, idep[i, x], HoldAllComplete] /@
Unevaluated@rhs // ReleaseHold]]];
idep[i_, _] := i;
When we run the tests we see that the last one failed runTest[dep]["TestsFailed"]
. That was expected. I don't know exactly how it makes sense to design this. The fact that associations don't become associations until they are evaluated, plus the fact that one has the possibility of holding any of its arguments through using :>
instead of ->
makes it confusing. Should one count as valid associations that aren't yet associations but are lexically well written? This does that, but this assumes that all valid unevaluated associations are Associations
with a list or sequence of rules inside, and doesn't check if those rules may have repeated keys which might end up deleting an element, and you can't really check without evaluating them and leaking.
Total
instead of 1 + Max
$\endgroup$
Here is another way of calculating the number of levels and elements. It first extracts the Values of each Association and then uses the resulting List to calculate the numbers. It would be nice to know if there is a simpler way of calculating the number of levels from the List.
levelsAndElements[data_Dataset] := Module[{lists, elements, levels},
lists = Normal@data //. assoc_Association :> Values[assoc];
elements = Length@Flatten[lists];
levels = Depth[DeleteCases[lists, Except[_List], Infinity]] - 1;
{levels, elements}]
levelsAndElements[ExampleData[{"Dataset", "Planets"}]]
{4, 350}