5
$\begingroup$

Note: the editor warns that this is a "subjective" post, but I am looking for an objective answer (e.g. in relation to easy of use of the many built-in functions, speed, etc).

Suppose you have a dataset which contains some hierarchical form and many other values which could be grouped by for analysis (copy paste into a notebook):

ds=Dataset[
 <|"Nest1_1" -> <|"Nest2_1" ->
     <|
      "Nest3_1" ->
       <|
        <|"Nest4_1" -> <|"a" -> 1, "b" -> 4, "c" -> 7|>|>,
        <|"Nest4_2" -> <|"a" -> 1, "b" -> 4, "c" -> 7|>|>
        |>,
      "Nest3_2" ->
       <|
        <|"Nest4_1" -> <|"a" -> 1, "b" -> 4, "c" -> 7|>|>,
        <|"Nest4_2" -> <|"a" -> 1, "b" -> 4, 
           "c" -> 7|>|>
        |>
      |>,
    "Nest2_2" ->
     <|
      "Nest3_1" ->
       <|
        <|"Nest4_1" -> <|"a" -> 1, "b" -> 4, "c" -> 7|>|>,
        <|"Nest4_2" -> <|"a" -> 1, "b" -> 4, "c" -> 7|>|>
        |>,
      "Nest3_2" ->
       <|
        <|"Nest4_1" -> <|"a" -> 1, "b" -> 4, "c" -> 7|>|>,
        <|"Nest4_2" -> <|"a" -> 1, "b" -> 4, 
           "c" -> 7|>|>
        |>
      |>

    |>,
  "Nest1_2" -> <|"Nest2_1" ->
     <|
      "Nest3_1" ->
       <|
        <|"Nest4_1" -> <|"a" -> 1, "b" -> 4, "c" -> 7|>|>,
        <|"Nest4_2" -> <|"a" -> 1, "b" -> 4, "c" -> 7|>|>
        |>,
      "Nest3_2" ->
       <|
        <|"Nest4_1" -> <|"a" -> 1, "b" -> 4, "c" -> 7|>|>,
        <|"Nest4_2" -> <|"a" -> 1, "b" -> 4, 
           "c" -> 7|>|>
        |>
      |>,
    "Nest2_2" ->
     <|
      "Nest3_1" ->
       <|
        <|"Nest4_1" -> <|"a" -> 1, "b" -> 4, "c" -> 7|>|>,
        <|"Nest4_2" -> <|"a" -> 1, "b" -> 4, "c" -> 7|>|>
        |>,
      "Nest3_2" ->
       <|
        <|"Nest4_1" -> <|"a" -> 1, "b" -> 4, "c" -> 7|>|>,
        <|"Nest4_2" -> <|"a" -> 1, "b" -> 4, "c" -> 7|>|>
        |>,
      "Nest3_3" ->
       <|
        <|"Nest4_1" -> <|"a" -> 1, "b" -> 4, "c" -> 7|>|>,
        <|"Nest4_2" -> <|"a" -> 1, "b" -> 4, 
           "c" -> 7|>|>
        |>
      |>

    |>
  |>]

which could also be flattened like:

Dataset[{
  <|"Nest1" -> 1, "Nest2" -> 1, "Nest3" -> 1, "Nest4" -> 1, "a" -> 1, 
   "b" -> 4, "c" -> 7|>,
  <|"Nest1" -> 1, "Nest2" -> 1, "Nest3" -> 1, "Nest4" -> 2, "a" -> 1, 
   "b" -> 4, "c" -> 7|>,
  <|"Nest1" -> 1, "Nest2" -> 1, "Nest3" -> 2, "Nest4" -> 1, "a" -> 1, 
   "b" -> 4, "c" -> 7|>,
  <|"Nest1" -> 1, "Nest2" -> 1, "Nest3" -> 2, "Nest4" -> 2, "a" -> 1, 
   "b" -> 4, "c" -> 7|>,
  <|"Nest1" -> 1, "Nest2" -> 2, "Nest3" -> 2, "Nest4" -> 1, "a" -> 1, 
   "b" -> 4, "c" -> 7|>,
  <|"Nest1" -> 1, "Nest2" -> 2, "Nest3" -> 2, "Nest4" -> 2, "a" -> 1, 
   "b" -> 4, "c" -> 7|>,
  <|"Nest1" -> 2, "Nest2" -> 2, "Nest3" -> 2, "Nest4" -> 1, "a" -> 1, 
   "b" -> 4, "c" -> 7|>,
  <|"Nest1" -> 2, "Nest2" -> 2, "Nest3" -> 2, "Nest4" -> 2, "a" -> 1, 
   "b" -> 4, "c" -> 7|>,
  <|"Nest1" -> 2, "Nest2" -> 2, "Nest3" -> 3, "Nest4" -> 1, "a" -> 1, 
   "b" -> 4, "c" -> 7|>,
  <|"Nest1" -> 2, "Nest2" -> 2, "Nest3" -> 3, "Nest4" -> 2, "a" -> 1, 
   "b" -> 4, "c" -> 7|>
  }]

Further lets pretend:

  • "Nest1" is some sort of condition (e.g. plant species)
  • "Nest2" is some sort of variable (e.g. given sunlight or not)
  • "Nest3" is some other sort of variable (e.g. amount watered)
  • "Nest4" is replicates of some measurement
  • "a","b","c" are some measurements (e.g. height, flowers, etc)

Which structure and why would be objectively better (in regards to leveraging the mechanics of the dataset object) for say comparing the average "height" ("a") across the two species ("Nest1") (without considering nests 2 and 3), as well as plotting the average number of flowers ("b") for each plant ("nest1") across each amount of water given ("nest3")?

Can example syntax be provided demonstrating the most native way to do this.

$\endgroup$
  • $\begingroup$ Maybe add a comparison of how you would answer your question(s) for each of the two dataset structures. If you need to extract different kind of groups from your data for different purposes, maybe it would be easier to start from the neutral flattened set ... $\endgroup$ – SquareOne Nov 22 '17 at 20:33
  • $\begingroup$ @Sum you picked really bad example data. First your numerical values are all the same, at least Randomize them. More importantly, your tree is balanced hiding the problem in doing iterated statistics over the groupings. This also lacks specificity, what "average" mean, median? Weighed, unweighed? There's dozens of averages. over what groups do you want to average "b"? $\endgroup$ – alancalvitti Feb 1 '18 at 18:29
  • $\begingroup$ If you modify your example w/ unbalanced grouping, where some groups have more elements (say leaf associations) than others, you'll see Mean[{a, b, c, d, e}] /. {a -> 10, b -> 20, c -> 30, d -> 40, e -> 50} from a serialized (ie, flat) table, doesn't equal the iterated averaging Mean[{Mean[{a, b}], Mean[{c, d, e}]}] /. {a -> 10, b -> 20, c -> 30, d -> 40, e -> 50} that you'd get in the nested approach using something like ds[All,Mean,Mean,Mean,"a"]. $\endgroup$ – alancalvitti Feb 1 '18 at 18:34
  • $\begingroup$ @alancalvitti I appreciate your well constructed criticism. You are right, my example is poorly construed thereby masking the issue. At the time I was more focused on ensuring this nested structure was viable. However, there may be another miscommunication. This isn't meant to be a tree. When asking my question, the data that inspired it was equally balanced and had multiple - equal - ways of ordering the values in the hierarchy. Likewise, the mean, was unspecific b/c for this data at the time, the average across all grouped combinations was needed. Nonetheless I may update this on the wknd $\endgroup$ – SumNeuron Feb 1 '18 at 18:59
  • $\begingroup$ @SumNeuron, the link in your profile was broken. I'm writing a short excursion section of my book on data to address your issue. If you want to get in touch, maybe we can itemize your issues. I think you started off with flat dataset and are asking how to compare statistics across pre-defined population groups. But would need to know more details. $\endgroup$ – alancalvitti Feb 6 '18 at 20:43
1
$\begingroup$

I support @alancalvitti observations. I've done loads of work with Al and will follow up his specific comments with the following. Al and I have worked on very large datasets (500G RAM), the kind and size one doesn't want to query SQL for many reasons. The first problem is never the 'shape' of the dataset it's the wrangling required of source data into a dataset one can understand and subsequently use efficiently for the intended purposes. Over the last couple years Al and I have zeroed in on a method of 'stepping' from loading source data that is not trusted and not knowable to a dataset that can be trusted and knowable.

When one has a dataset of hundreds of keys, sometimes 12 to 15 levels deep, with nearly 1B values, jagged, and missing keys and values all over the place, and one wants to generate a model... well... The long and short of my experience of using Dataset and Association on those scales is, once you get the wrangling right, a large query on a large trusted dataset is shockingly efficient. I think it's all in the data preparation for the purpose of analysis. What does the 'shape' the data need to be to accomplish the analysis? I guess this is a long way of saying, I think there is no optimal dataset 'shape' , but there is an optimal 'shape' an analysis wants the data to be. So, going back to my first comments, wrangle the data to a state of trustworthiness, then shape the data to a specific analysis. Along the way you will have several iterations 'steps' the data will go through to accomplish your need.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.