The Stack Overflow podcast is back! Listen to an interview with our new CEO.
1 of 2

With a lot of help from george2079 and Jack LaVigne, I have arrived at a solution to my issue.

  1. Part of my problem was the syntax for my pure functions
  2. Additionally, GatherBy does not order the subgroups in a "sorted" way.
  3. I couldn't "distribute" a pure function Sort over a list of lists.

Given these considerations, something like

GatherBy[Sort[rings[3],Lmag@Coefficient[#1, {q1, q2, q3}] < Lmag@Coefficient[#2, {q1, q2, q3}] &], Lmag@Coefficient[#, {q1, q2, q3}] &];  

Will give the proper subsets ordered by their Norms. First, you must Sort the list rings[3] by magnitude, and then GatherBy to separate the subsets. For some reason, if you don't sort them first, GatherBy will arrange the subsets in order of the first appearance of each equivalence class! This behavior is unusual, because GatherBy does not work this way for the Numerical case!

Now, I don't have mma v10 which includes a similar command GroupBy (I actually can't tell how it differs from GatherBy) so there is a chance, I suppose, that function might accomplish both goals simultaneously.

In the numeric case, I could then simply "distribute" a Sort over these lists of lists to sort by the ArcTan function. Perhaps if I were more adept at pure functions, this could still be done in a single step, I'm not sure. Nonetheless, it can be done by simply sorting each sublist individually, which is fine for my purpose.

With that, should anyone come along and care to see it, here is a fully functional solution to my original post

Block[{order = 4, rings, splitrings, stars, q1, q2, q3}, rings[0] = {0}; stars[0, 0] = rings[0]; rings[1] = {q1, q2, q3}; Do[rings[r] = Complement[
Flatten[Table[rings[r - 1][[s]] + (-1)^(r - 1) Boole[\[Sigma] ==1] q1 +(-1)^(r - 1) Boole[\[Sigma] ==2] q2 + (-1)^(r - 1) Boole[\[Sigma] == 3] q3, {s, 1,Length[rings[r - 1]]}, {\[Sigma], 1, 3}], 1], 
rings[r - 2]], {r, 2, order}]; Do[splitrings[r] = GatherBy[   Sort[rings[r], Lmag@Coefficient[#1, {q1, q2, q3}] < 
   Lmag@Coefficient[#2, {q1, q2, q3}] &], 
Lmag@Coefficient[#, {q1, q2, q3}] &], {r, 1, order}]; Do[stars[r, s] =    Sort[splitrings[r][[s]], L\[CurlyPhi]@Coefficient[#1, {q1, q2, q3}] < 
  L\[CurlyPhi]@Coefficient[#2, {q1, q2, q3}] &], {r, 1, order}, {s, 1, Length@splitrings[r]}]]