How to create dataset with ordered and hierarchically-grouped rows?

Question

The (toy-example) dataset plots consists of 12 rows, the first two of which are shown below:

I want to arrange the plots shown in the plot column into a grid, where the rows correspond to values of the color and suit variables, and the columns correspond to the values of the elf variable.

More specifically, the rows should be grouped into two "superrows", one for each value of the color variable, in the order {"red", "black"}, and the rows within each superrow should correspond to the values of the suit variable, ordered alphabetically.

I've only managed to achieve a simple grid, where the rows are neither grouped nor ordered:

Here's the code that does it:

grid = plots[GroupBy["suit"], GroupBy[KeyDrop[#, "suit"], Key["elf"], Function[x, First[x][["plot"]]]] &];

(I'd be embarrassed to admit how many hours it took me to conquer this molehill.)

---

My questions are:

1. How can I create the superrows?
2. How can I display the labels of the rows and superrows as two hierarchical header columns along the left?
3. How can I order the two levels as described above?

---

FWIW, below is the code that generates the plots dataset.

makeDataset[columns_, columnLabels_] := Module[{labeledColumns, records},
   labeledColumns = AssociationThread[columnLabels, columns];
   records = Transpose[labeledColumns, AllowedHeads -> All];
   Dataset[records]
];

preplots = Module[{points, data, base, columns, columnNames, validCombinationPattern, validRows},
   points = Flatten[Table[Table[Chop[{Sin[m (u + 2)], Sin[n u]}], {u, 0, 4 \[Pi], 0.1}],
                          {n, 4}, {m, 3}], 1];
   data = Map[makeDataset[Transpose[#], {"x", "y"}] &, points];

   base = <|"suit" -> {"Spades", "Hearts", "Diamonds", "Clubs"},
            "color" -> {"Black", "Red"},
            "elf" -> {"Snap", "Crackle", "Pop"}|>;

   validCombinationPattern = {"Spades" | "Clubs", "Black", _} | {"Hearts" | "Diamonds", "Red", _};
   validRows = Cases[Tuples[Values[base]], validCombinationPattern];
   columns = Append[Transpose[validRows], data];
   columnNames = Append[Keys[base], "data"];

   makeDataset[columns, columnNames]
];

doPlot[dataset_] := ListPlot[dataset[["data"]], AspectRatio -> 1, Frame -> True, FrameTicks -> None];
plots = preplots[All, KeyDrop[Append[#, "plot" -> doPlot[#]], "data"] &];

---

Update

Here's the seed of an idea regarding the problem (mentioned by WReach) of the repeated column headers.

One can use the part spec [[All, Values]] on the 2nd-level datasets to effectively suppress the column headers:

The code for that is the following modification of WReach's pipeline:

plots[
  GroupBy[{#color&, #suit&, #elf& -> (#plot&)}] /* KeySortBy[<|"Red"->1,"Black"->2|>]
, KeySort /* KeyMap[Pane[#, 50] &] /* (Dataset[#][[All, Values]] &)
, KeySort
, First
]

(Actually, for the figure shown, I had to use a width of 70 rather than 50 for the suit header. YMMV.)

What I still have not figured out how to do is to apply this modification selectively to all but the first superrow.

---

Lingering minor gripe: the 2 levels | 2 rows, &c. schmutz at the bottom of the dataframes is an unwelcome distraction. I wish one could suppress it somehow. To WRI: This ... levels | ... rows information is of interest only for debugging; once the dataset's structure is as it should be, and its contents are clearly displayed, one couldn't care less about the ... levels | ... rows information. Please add "remove levels/rows annotation from default Dataset display" to the wishlist for 10.5.

Answer 1

Update - Version 11

In version 11, The Dataset visualizer handles triply-nested associations quite nicely:

plots[
  GroupBy[{#color&, #suit&, #elf& -> (#plot&)}] /* KeySortBy[<|"Red"->1, "Black"->2|>]
, KeySort
, KeySort
, First
]

---

Original Response

The Dataset visualizer does not presently (v10.4.1) support a nice layout for triply-nested associations. However, we can get close by nesting datasets:

plots[
  GroupBy[{#color&, #suit&, #elf& -> (#plot&)}] /* KeySortBy[<|"Red"->1,"Black"->2|>]
, KeySort /* Dataset
, KeySort
, First
]

KeySort and KeySortBy are used at each association level to order the keys.

This rendition is not perfect, but has the virtue of being achievable with relatively simple code. We can get better alignment of the columns by using Pane to fix the width of the suit row headers:

plots[
  GroupBy[{#color&, #suit&, #elf& -> (#plot&)}] /* KeySortBy[<|"Red"->1,"Black"->2|>]
, KeySort /* KeyMap[Pane[#, 50]&] /* Dataset
, KeySort
, First
]

The repeated headers are harder to fix. It requires, for example, creating a new visualization from scratch using Grid (left as an exercise for the reader :).

Bug In Mathematica 10.2 and Earlier

In Mathematica version 10.2, a bug in the Dataset type system will prevent the exhibited expressions from working properly. To dodge the bug, we must use the usual work-around when faced with a type-error in Dataset: extract the data from the Dataset, transform it, and then stuff it back into a new Dataset:

plots// Normal // Query[
  GroupBy[{#color&, #suit&, #elf& -> (#plot&)}] /*
    KeySortBy[<|"Red"->1,"Black"->2|>] /* Dataset
, KeySort /* KeyMap[Pane[#, 50]&] /* Dataset
, KeySort
, First
]