# Dataset: get number of levels and elements

The summary box for Dataset objects shows the number of levels and elements at the bottom.

How can we get these values programmatically?

For example, "4 levels, 350 elements" in the Planets dataset:

If we are not afraid of using undocumented functions, we can get the counts the same way that the summary box gets them:

info @ Dataset[data_, type_, _] :=
{TypeSystemTypeDepth[type],  TypeSystemAtomCount[type, data]}

info @ ExampleData[{"Dataset", "Planets"}]
(* {4, 350} *)


Or, as per @Szabolcs' suggestion:

info2 @ ds_Dataset :=
{TypeSystemTypeDepth[#],  TypeSystemAtomCount[##]}& @@
Through[{DatasetGetType, DatasetGetData} @ ds]

info2 @ ExampleData[{"Dataset", "Planets"}]
(* {4, 350} *)

• The old master strikes again. :D ("Old" only respectfully, and in the sense that you were the first serious Mathematica user on Stack Exchange.) – Mr.Wizard Aug 10 '14 at 5:07
• I spent a few of minutes looking at the functions in TypeSystem and couldn't find them. Grrrr +1 – Rojo Aug 10 '14 at 5:14
• I think the "official" way to get the type is DatasetGetType[] (still undocumented, of course). – Szabolcs Aug 10 '14 at 5:18
• @Szabolcs I added a new variation incorporating your suggestion. – WReach Aug 10 '14 at 13:39

This is not quite the same thing, but strongly related and very useful to know:

It appears that the undocumented option AllowedHeads, which appears in several functions in v10, can help here.

ds = ExampleData[{"Dataset", "Planets"}];


To get the depth,

ArrayDepth[
Normal[ds],
]


This also works:

ArrayDepth[
Normal[ds],
]


To get the element count from this generalized array,

Times @@ Dimensions[
Normal[ds],
]


These counts refer only to the rectangular part of the generalized array made of lists and associations. They won't give the counts 4 and 350. For that we'd need to descend deeper.

Out of curiosity, this is the complete list of System functions having this option:

ArrayDepth, ConjugateTranspose, Depth, Dimensions, Transpose


Transpose and ConjugateTranspose are useful for taking the transpose of mixed List-Association structures.

Depth would theoretically be good for finding the number of levels in the dataset, but I don't know how to use this option with it. It seems to take only the values True and False.

• ArrayDepth is same as Dimensions /* Length. But Dimensions itself is more informative for many data processing tasks. – alancalvitti Oct 7 '14 at 18:49

There's at least the brute force approach

getFromDs[what_String][ds_Dataset] :=
FirstCase[ToBoxes@ds,
RowBox[{___, i_, Shortest@___, what, ___}] :>
With[{res = ToExpression@i}, res /; IntegerQ[res]], \$Failed,
Infinity]

getLevels = getFromDs["levels"]; getElements = getFromDs["elements"];

Through@{getLevels, getElements}@ExampleData[{"Dataset", "Planets"}]

(* {4, 350} *)


This is an attempt at a disgusting function for the depth: the maximum number of lists or associations you can go through from the top down before bouncing in something else

We setup some tests

test[dep] ^= Inactivate@{
dep[{1}] -> 1,
dep[4] -> 0,
dep[h[4, g[8]]] -> 0,
dep[<|"a" -> 3|>] -> 1,
dep[<|2|>] -> 0,
dep[<|"a" -> 2|>] -> 1,
dep[{<|"a" -> 2|>, d}] -> 2,
dep[<|{2}|>] -> 0,
dep[<|"a" -> {2}, "b" -> 8|>] -> 2,
dep[<|"a" :> {<|"c" -> h[2]|>}|>] -> 3,
dep[<|"x" -> {1}, "x" -> 3|>] -> 1,
dep[<|"x" :> <|"x" -> {1}, "x" -> 3|>|>] -> 2
};

runTest[sym_] :=
Inactive[TestReport][
test[sym] /. (x_ -> y_) :> Inactive[VerificationTest][x, y]] //
Activate


The code

dep[ds_Dataset] := dep[Normal@ds];
dep[stuff_] := idep[0, stuff];

SetAttributes[idep, HoldAllComplete];
idep[i_, l_List |
l_Association?(Function[i, AssociationQ@Unevaluated@i,
HoldAllComplete])] :=
1 + Max[Function[x, idep[i, x], HoldAllComplete] /@
Unevaluated@l /. {a_Association :> Values@a}];
idep[i_, a_Association /;
MatchQ[Unevaluated@a,
HoldPattern[
Association[(_Rule | _RuleDelayed) ... | {(_Rule | \
_RuleDelayed) ...}]]]] :=

Module[{v = Block[{Association = HoldComplete}, a]},
With[{rhs = v[[All, 2]]},
1 + Max[Function[x, idep[i, x], HoldAllComplete] /@
Unevaluated@rhs // ReleaseHold]]];
idep[i_, _] := i;


When we run the tests we see that the last one failed runTest[dep]["TestsFailed"]. That was expected. I don't know exactly how it makes sense to design this. The fact that associations don't become associations until they are evaluated, plus the fact that one has the possibility of holding any of its arguments through using :> instead of -> makes it confusing. Should one count as valid associations that aren't yet associations but are lexically well written? This does that, but this assumes that all valid unevaluated associations are Associations with a list or sequence of rules inside, and doesn't check if those rules may have repeated keys which might end up deleting an element, and you can't really check without evaluating them and leaking.

• One could probably get the number of elements similary, taking the Total instead of 1 + Max – Rojo Aug 9 '14 at 4:09

Here is another way of calculating the number of levels and elements. It first extracts the Values of each Association and then uses the resulting List to calculate the numbers. It would be nice to know if there is a simpler way of calculating the number of levels from the List.

levelsAndElements[data_Dataset] := Module[{lists, elements, levels},
lists = Normal@data //. assoc_Association :> Values[assoc];
elements = Length@Flatten[lists];
levels = Depth[DeleteCases[lists, Except[_List], Infinity]] - 1;
{levels, elements}]

levelsAndElements[ExampleData[{"Dataset", "Planets"}]]

{4, 350}