12
$\begingroup$

The summary box for Dataset objects shows the number of levels and elements at the bottom.

How can we get these values programmatically?

For example, "4 levels, 350 elements" in the Planets dataset:

Mathematica graphics

$\endgroup$

4 Answers 4

11
$\begingroup$

If we are not afraid of using undocumented functions, we can get the counts the same way that the summary box gets them:

info @ Dataset[data_, type_, _] :=
  {TypeSystem`TypeDepth[type],  TypeSystem`AtomCount[type, data]}

info @ ExampleData[{"Dataset", "Planets"}]
(* {4, 350} *)

Or, as per @Szabolcs' suggestion:

info2 @ ds_Dataset :=
  {TypeSystem`TypeDepth[#],  TypeSystem`AtomCount[##]}& @@
    Through[{Dataset`GetType, Dataset`GetData} @ ds]

info2 @ ExampleData[{"Dataset", "Planets"}]
(* {4, 350} *)
$\endgroup$
4
  • $\begingroup$ The old master strikes again. :D ("Old" only respectfully, and in the sense that you were the first serious Mathematica user on Stack Exchange.) $\endgroup$
    – Mr.Wizard
    Aug 10, 2014 at 5:07
  • $\begingroup$ I spent a few of minutes looking at the functions in TypeSystem and couldn't find them. Grrrr +1 $\endgroup$
    – Rojo
    Aug 10, 2014 at 5:14
  • $\begingroup$ I think the "official" way to get the type is Dataset`GetType[] (still undocumented, of course). $\endgroup$
    – Szabolcs
    Aug 10, 2014 at 5:18
  • $\begingroup$ @Szabolcs I added a new variation incorporating your suggestion. $\endgroup$
    – WReach
    Aug 10, 2014 at 13:39
9
$\begingroup$

This is not quite the same thing, but strongly related and very useful to know:

It appears that the undocumented option AllowedHeads, which appears in several functions in v10, can help here.

ds = ExampleData[{"Dataset", "Planets"}];

To get the depth,

ArrayDepth[
 Normal[ds],
 AllowedHeads -> {Association, List}
]

This also works:

ArrayDepth[
 Normal[ds],
 AllowedHeads -> All
]

To get the element count from this generalized array,

Times @@ Dimensions[
  Normal[ds],
  AllowedHeads -> All
]

These counts refer only to the rectangular part of the generalized array made of lists and associations. They won't give the counts 4 and 350. For that we'd need to descend deeper.


Out of curiosity, this is the complete list of System` functions having this option:

ArrayDepth, ConjugateTranspose, Depth, Dimensions, Transpose

Transpose and ConjugateTranspose are useful for taking the transpose of mixed List-Association structures.

Depth would theoretically be good for finding the number of levels in the dataset, but I don't know how to use this option with it. It seems to take only the values True and False.

$\endgroup$
1
  • $\begingroup$ ArrayDepth is same as Dimensions /* Length. But Dimensions itself is more informative for many data processing tasks. $\endgroup$ Oct 7, 2014 at 18:49
5
$\begingroup$

There's at least the brute force approach

getFromDs[what_String][ds_Dataset] := 
 FirstCase[ToBoxes@ds, 
  RowBox[{___, i_, Shortest@___, what, ___}] :> 
   With[{res = ToExpression@i}, res /; IntegerQ[res]], $Failed, 
  Infinity]

getLevels = getFromDs["levels"]; getElements = getFromDs["elements"];

Through@{getLevels, getElements}@ExampleData[{"Dataset", "Planets"}]

(* {4, 350} *)

This is an attempt at a disgusting function for the depth: the maximum number of lists or associations you can go through from the top down before bouncing in something else

We setup some tests

test[dep] ^= Inactivate@{
    dep[{1}] -> 1,
    dep[4] -> 0,
    dep[h[4, g[8]]] -> 0,
    dep[<|"a" -> 3|>] -> 1,
    dep[<|2|>] -> 0,
    dep[<|"a" -> 2|>] -> 1,
    dep[{<|"a" -> 2|>, d}] -> 2,
    dep[<|{2}|>] -> 0,
    dep[<|"a" -> {2}, "b" -> 8|>] -> 2,
    dep[<|"a" :> {<|"c" -> h[2]|>}|>] -> 3,
    dep[<|"x" -> {1}, "x" -> 3|>] -> 1,
    dep[<|"x" :> <|"x" -> {1}, "x" -> 3|>|>] -> 2
    };

runTest[sym_] := 
 Inactive[TestReport][
   test[sym] /. (x_ -> y_) :> Inactive[VerificationTest][x, y]] // 
  Activate

The code

dep[ds_Dataset] := dep[Normal@ds];
dep[stuff_] := idep[0, stuff];

SetAttributes[idep, HoldAllComplete];
idep[i_, l_List | 
    l_Association?(Function[i, AssociationQ@Unevaluated@i, 
       HoldAllComplete])] := 
  1 + Max[Function[x, idep[i, x], HoldAllComplete] /@ 
      Unevaluated@l /. {a_Association :> Values@a}];
idep[i_, a_Association /; 
    MatchQ[Unevaluated@a, 
     HoldPattern[
      Association[(_Rule | _RuleDelayed) ... | {(_Rule | \
_RuleDelayed) ...}]]]] :=

  Module[{v = Block[{Association = HoldComplete}, a]},
   With[{rhs = v[[All, 2]]}, 
    1 + Max[Function[x, idep[i, x], HoldAllComplete] /@ 
        Unevaluated@rhs // ReleaseHold]]];
idep[i_, _] := i;

When we run the tests we see that the last one failed runTest[dep]["TestsFailed"]. That was expected. I don't know exactly how it makes sense to design this. The fact that associations don't become associations until they are evaluated, plus the fact that one has the possibility of holding any of its arguments through using :> instead of -> makes it confusing. Should one count as valid associations that aren't yet associations but are lexically well written? This does that, but this assumes that all valid unevaluated associations are Associations with a list or sequence of rules inside, and doesn't check if those rules may have repeated keys which might end up deleting an element, and you can't really check without evaluating them and leaking.

$\endgroup$
1
  • $\begingroup$ One could probably get the number of elements similary, taking the Total instead of 1 + Max $\endgroup$
    – Rojo
    Aug 9, 2014 at 4:09
1
$\begingroup$

Here is another way of calculating the number of levels and elements. It first extracts the Values of each Association and then uses the resulting List to calculate the numbers. It would be nice to know if there is a simpler way of calculating the number of levels from the List.

levelsAndElements[data_Dataset] := Module[{lists, elements, levels},
  lists = Normal@data //. assoc_Association :> Values[assoc];
  elements = Length@Flatten[lists];
  levels = Depth[DeleteCases[lists, Except[_List], Infinity]] - 1;
  {levels, elements}]

levelsAndElements[ExampleData[{"Dataset", "Planets"}]]
{4, 350}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.