Okay, new approach. My old answer is preserved below for reference.
I was not aware of this before posting, but there is a
MenuPosition option in the Options Inspector, and it does take effect. You will need to first check
Editable so that you can edit the palette. Here is the active screen:
In the header of a palette .nb file there is this section:
(* Internal cache information:
NotebookDataPosition[ 145, 7]
NotebookDataLength[ 28489, 713]
NotebookOptionsPosition[ 27870, 688]
NotebookOutlinePosition[ 28422, 710]
CellTagsIndexPosition[ 28379, 707]
The value of
MenuPosition, if present determines the group and ordering of the palette in the menu. That is, lower values appear higher in the list, and palettes with the same value will be placed in the same group.