I have put to the test Leonid's second suggestion, ApplicationMaker by jmlopez, I have shared out an extended example here. It's a little library for rotation quaternions and rigid-body dynamics that illustrates the Dzhanybekhov effect. I created it from scratch by straight reading of the documents provided with ApplicationMaker (most of it in a very nice local cigar lounge, but that's another story :) Here are my major findings:
You do NewPackage and get a notebook for writing your package (ApplicationMaker will make the .m file for you from this notebook). Don't get even slightly fancy with usage messages in here! In my first attempt, I tried something as little as setting the arguments for function forms in italics, and it caused a cascade of crap to happen later when I did BuildApplication and DeployApplication. I had to rekeyboard all my ::usage messages because even one stray invisible Italic tag caused massive downstream pain -- missing styles, all kinds of StyleBox noise copies to the reference pages, many more goobers I didn't even look into.
Other than that, it seemed to work well with MMA 9, even though it was written for MMA 8. Huge kudos to jmlopez for working this out. While it's not perfect, it's such a great start that I will certainly use it in the future.
I did not complete the documentation for my sample in the interests of getting this answer out quickly. Another couple of findings (really reminders for me) are:
Proper documentation is a LOT of work. It easily outstrips the effort of writing the code by factors of 2 to 10. Even for my tiny example, it would probably take me several more days to get it to professionally presentable level.
HOWEVER it is very valuable work! The effort of writing documentation and thinking about how someone else would try to understand this tiny library made me improve it by refactoring and redesigning several times. A professionally presentable API isn't even fully DESIGNED before it's DOCUMENTED with a skeptical, third-party audience in mind! Just do it.