In my experience, the rule of thumb is that the generated code will be efficient if the code inside `Compile` more or less resembles the code I would write in plain C (and it is clear why). Idiomatic (high-level) Mathematica code tends to be immutable. At the same time, `Compile` can handle a number of higher-level functions, such as `Transpose`, `Partition`, `Map`, `MapThread`, etc. So, if you write your code in this style and hand it to `Compile`, you have to keep in mind that lots of small (or large) memory allocations on the heap, and copying of lists (tensors) will be happening. 

Since this is not apparent for someone who is used to high-level Mathematica programming, the slowdown this may incur may be surprising (one example of problems coming from many small memory allocations and copying, as well as a speed-up one can get from switching from copying to in-place modifications, can be found in [this answer][1]. Another example of the speed-up gains from replacing copying with in-place modifications can be found in my answer to [this question][2])  

As noted by @acl, one thing worth doing is to set the `SystemOptions -> "CompileOptions"` as

    SetSystemOptions[ "CompileOptions" -> "CompileReportExternal" -> True]

in which case you will get warnings for calling external functions etc.

A good tool to get a "high-level" but precise view on the generated code is the `CompilePrint` function in the ``CompiledFunctionTools` `` package. It allows you to print the pseudocode version of the byte-code instructions generated by `Compile`. Things to watch for in the printout of `CompilePrint` function:

 - Calls to `CopyTensor`
 - Calls to MainEvaluate (callbacks to Mathematica, meaning that something could not be compiled down to C)

One not very widely known technique of writing even large `Compile`-d functions and combining them from pieces so that there is no performance penalty, is based on inlining. I consider [this answer][3] very illustrative in this respect - I actually posted it to showcase the technique. You can also see [this answer][4] and a discussion in the comments below, for another example of how this technique may be applied.  

So, in summary - if you want your code to be as fast as possible, think about "critical" places and write those in "low-level" style (loops, assignments, etc) - the more it will resemble C the more chances you have for a speed-up (for an example of a function written in such a style and being consequently very fast, see the `seqposC` function from [this answer][5]). You will have to go against Mathematica ideology and use a lot of in-place modifications. Then your code can be just as fast as hand-written one. Usually, there are just a few places in the program where this matters (inner loops, etc) - in the rest of it you can use higher-level functions as well.


  [1]: http://stackoverflow.com/questions/8183501/efficient-handling-of-list-of-small-vectors-with-compile/8184350#8184350
  [2]: http://stackoverflow.com/questions/5771217/how-to-efficiently-set-matrixs-minor-in-mathematica
  [3]: http://stackoverflow.com/questions/5246330/delete-repeating-list-elements-preserving-order-of-appearance/5251034#5251034
  [4]: http://stackoverflow.com/questions/1165742/yield-in-mathematica/5705670#5705670
  [5]: http://stackoverflow.com/questions/8364804/a-fast-implementation-in-mathematica-for-position2d/8367453#8367453