VertexNormals depends on Polygon's orientation - an expected behaviour?

Question

I'm studying VertexNormals (on Mathematica 10.0.2 for Mac) and have set up the following to represent three points and a normal vector:

p = {{1, 1, 0}, {1, 0, 1}, {0, 1, 1}};
n = {1, 1, 1};

and the following function to display a triangle whose vertices all have n as their normal direction:

g[p_] := Graphics3D[
  {
    Yellow, Polygon[p, VertexNormals -> {n, n, n}],
    MapThread[{#1, Sphere[#2, 0.1]} &, {{Red, Green, Blue}, p}]
  },
  Axes -> True, AxesLabel -> {x, y, z}, ViewPoint -> 100 {1, 1, 1}
];

The first vertex is marked by a red sphere, the second with a green one, and the third with a blue one.

g[p] produces

whereas g[Reverse@p] produces

Clearly, the normal direction actually depends on the orientation of the polygon. Is this an expected behaviour? I find this to be against intuition. If for example I specify that the normal direction for a particular vertex is along $\hat\imath + \hat\jmath + \hat{k}$, I'd rather expect that it always points away from the origin regardless of where other vertices are. This does not seem to be the case, and the documentation for VertexNormals doesn't say anything about this.

Could anyone please enlighten me as to why VertexNormals is designed to behave as such?

And given that such behaviour can't change, what's the computationally cheap way of choosing the right direction for vertex normals? The cheapest one that I can think of is, as described in the comment,

g[p_] := Graphics3D[
  {
    Yellow, Polygon[p, VertexNormals -> If[
      VectorAngle[Cross[p[[2]] - p[[1]], p[[3]] - p[[1]]], n] < Pi/2,
      1,
      -1
    ]{n, n, n}],
    MapThread[{#1, Sphere[#2, 0.1]} &, {{Red, Green, Blue}, p}]
  },
  Axes -> True, AxesLabel -> {x, y, z}, ViewPoint -> 100 {1, 1, 1}
];

Answer 1

As noted in the docs, the only effect of specifying VertexNormals is in the shading; recall that Mathematica uses the Phong model for depicting surfaces. The docs also have a review of how shading is done in Mathematica.

For further illustrations, here is a modified version of your demonstration function g:

g[p_, nl_, opts___] := Graphics3D[{FaceForm[Yellow, Brown],
                                   Polygon[p, VertexNormals -> nl],
                                   {Arrowheads[Small],
                                    MapThread[{#1, Arrow[Tube[{#2,
                                                               #2 + Normalize[#3]/4}]]} &,
                                              {{Red, Green, Blue}, p, nl}]}}, opts,
                                  Axes -> True, AxesLabel -> {"x", "y", "z"},
                                  ViewPoint -> 100 {1, 1, 1}]

I have mentioned this a number of times before on this site, but let me say it again: FaceForm[] is a very valuable visual diagnostic tool for determining the orientation of the polygons in your graphic. One merely needs to give strikingly contrasting directives as its first and second arguments to see if there are any wayward polygons in your surface. In this situation, we use it to confirm the orientation of your triangle in relation to the normals.

Here is the equivalent of your two pictures:

{g[p, {n, n, n}], g[Reverse[p], {n, n, n}]} // GraphicsRow

This demonstrates that it is the order of the points in p that determines the color, and that it is oriented clockwise when facing the viewer.

The following two sets of pictures show the effect of varying the normal orientations for both the clockwise and anticlockwise cases:

Map[g[p, # {n, n, n}, PlotLabel -> #] &, 
    Partition[Tuples[{1, -1}, {3}], 4], {2}] // GraphicsGrid

Map[g[Reverse[p], # {n, n, n}, PlotLabel -> #] &, 
    Partition[Tuples[{1, -1}, {3}], 4], {2}] // GraphicsGrid

---

It would seem that you've already mentioned in the comments the method on how to generate a normal that is consistent with your polygon's orientation. Here is a demo:

Show[g[Reverse[p], {n, n, n}], 
     With[{c = Mean[p], 
           n = Normalize[Cross[#2 - #1, #3 - #1]] & @@ Reverse[p]}, 
          Graphics3D[{Arrowheads[Medium], Black, Arrow[{c, c + n/2}]}]], 
     ViewPoint -> {100, 20, 100}]