You basically want to permute the elements of the list among some specified set of permutations, and find the ordering of the list among these permutations that is canonically "first". The set of permutations you're interested in is a subgroup of the permutation group $S_n$, generated by two generators, one of which rotates all elements one position to the left and the other of which reverses the elements.
We can use Mathematica's built-in group theory functions to do this. PermutationGroup
specifies a group generated by any number of permutations, and GroupElements
tells Mathematica to explicitly calculate all the elements of this group. We can then apply all of these permutations to the original list, apply Sort
to that, and then take the first element of the sorted list of permuted lists. Like so:
rotrevsort[listin__] :=
Module[{list = listin, n, permlist},
n = Length[list];
permlist = GroupElements[PermutationGroup[{PermutationCycles[RotateLeft[Range[n]]],
PermutationCycles[Reverse[Range[n]]]}]];
First[Sort[Permute[list, #] & /@ permlist]]
]
rotrevsort[{c, b, d, a}]
rotrevsort[{d, a, b, a}]
rotrevsort[{0, b, a, 0}]
(* {a, c, b, d} *)
(* {a, b, a, d} *)
(* {0, 0, a, b} *)
By the way, your group is only of order $2n$, i.e., there are only $2n$ possible permutations of the list that you need to sort. (For those who know group theory, the subgroup of $S_n$ we're looking at is isomorphic to the dihedral group $D_n$.) So if we assume that Sort
's algorithms don't take significantly longer to sort a list of $2n$ elements than they do a list of $n$ elements, I wouldn't expect that sorting the whole list of valid permutations would be significantly less efficient than sorting the elements of the list themselves.