In the course of answering this question, I ran into a little bit of weirdness that doesn't square with my experience with previous versions of Mathematica. I think writing this answer is as good a time as any to bring it up.
Firstly, there is this tantalizing line from the internal implementation notes:
FunctionExpand uses an extension of Gauss's algorithm to expand trigonometric functions with arguments that are rational multiples of $\pi$.
Indeed, in the old versions of Mathematica,
FunctionExpand certainly was able to convert trigonometric values at rational multiples of $\pi$ to the corresponding radical values (possibly involving complex numbers). This hinged on the function
Developer`TrigToRadicals, and in fact one could use this directly instead of
FunctionExpand no longer seems to perform this conversion. Using
Developer`TrigToRadicals throws an error message saying that it is obsolete, and that one should now use
ToRadicals instead. The kicker is that
Developer`TrigToRadicals return different-looking (but numerically equivalent) results, which means the internal algorithms have been changed somewhat.
Quite a mystery.
Now, "Gauss's algorithm" is not terribly informative, considering that Gauss was a rather prodigious producer of mathematical results. However, there is a section in Gauss's Disquisitiones Arithmeticae (sorry, I couldn't find a free English translation) that relates to a number-theoretic view of the roots of unity (and thus, trigonometric functions). In particular, he outlines in one section a recursive method for determining explicit radical expressions of roots of unity.
There is, however, a more modern method that improves on the asymptotic complexity of Gauss's algorithm. In this paper, A. Weber presents an improvement of Gauss's original algorithm that hinges on an efficient way to evaluate the "recursive step" in the original method. A Maple implementation of this improved method is presented in the paper; I will wager that this method, with possibly a few proprietary extensions, is the guts behind
Developer`TrigToRadicals. (It should be noted that Weber's paper also points to alternative algorithms for explicitly producing radical expressions; I have not been able to find those references, so I cannot say much about them. Maybe an expert can chime in.)
As for the original question, although $\sin(\pi/5)$ is automatically converted to a radical result in the current version of Mathematica, it is likely that the improved Gauss algorithm was used to produce the radical expression in the first place.