How can I set the view vector in a 3D scene by tracking my face's position?

Question

Faking 3D with 2D tracking data is something that I've long wanted to implement for use with my front facing laptop camera. A first step to solving this would be dynamically setting the viewpoint according to the movement of your head.

Edit:

To clarify, I'm looking for an answer like the code I'm giving below. The reason I ask is because there were a number of problems with my own solution:

1. The method used is not sensitive enough
2. It is far too slow
3. Not robust to background movement

So a real answer to this question might use some fancy bit of machine vision, or interface with opencv face tracking libraries, or something else entirely. Here is a video of this technique applied sucessfully: http://www.youtube.com/watch?v=7saRtCUgbNI

ClusterMean[x_, 0, _] := Mean[x];
ClusterMean[{x_}, n_, frac_] := x;
ClusterMean[x_, n_, frac_] := 
  With[{m = Mean[x], sd = StandardDeviation[x]}, 
   With[{x2 = Select[x, Norm[Norm[# - m]/sd] < frac &]}, 
    If[x2 == {}, Mean[x], ClusterMean[x2, n - 1, frac]]]];
$ytrim = $xtrim = {1, All};
SmallCurrentImage[] := ImageTake[CurrentImage[], $ytrim, $xtrim];

$first = SmallCurrentImage[];

FaceOffset[center : {_, _}] :=

 Module[{mean, img1, img2, matches}, mean = {0, 0};
  img1 = $first;
      $currentimage = img2 = SmallCurrentImage[];
  $matches = matches = ImageCorrespondingPoints[img1, img2];
  mean = MapThread[Subtract, matches];

  If[mean == {}, center, ClusterMean[mean, 1, 0.8]]
  ]

center = {0, 0}; center2 = {0, 0}; $first = SmallCurrentImage[];

Dynamic[
 If[CurrentValue["MouseOver"], center = FaceOffset[center];
  center2 = center2*0.5 + center*0.5;];

 Refresh[
  Labeled[
   Refresh[
    Graphics3D[{PolyhedronData["RhombicHexecontahedron", 
       "Faces"]}, 
     ViewPoint -> 
      Refresh[5*Normalize[Prepend[-center2/80, -1]], 
       TrackedSymbols :> {center}], SphericalRegion -> True, 
     RotationAction -> "Clip"], 
    None], {Show[$currentimage, 
         Graphics[{Red, MapThread[Line[{#1, #2}] &, $matches]}]], 
        Button["Reset View", ($first = SmallCurrentImage[]; 
      center2 = {0, 0};), Appearance -> "Palette", 
     ImageSize -> Automatic]}, {Left, {Bottom, Right}}], 
  TrackedSymbols :> {center}]]

Answer 1

You might try to expand the example below for a more generic solution. This function uses the webcam to track a red object and updates the viewpoint according to the position of this object. You draw a nice red bindi on your forehead, move the head left or right and the viewpoint will update as a function of your head position. Right now it works for left/right translations only.

  activeVision[shape_] :=
  Dynamic[
   With[{img = CurrentImage[]},
    Graphics3D[
     shape,
     SphericalRegion -> True, Boxed -> False, AspectRatio -> 1, 
     Background -> Black, ViewPoint -> RotationTransform[-N[ArcTan[
            (Mean[
                 ComponentMeasurements[
                   Threshold[
                    ImageSubtract[ImageSubtract[#1, #2], #3] &  @@ 
                    ColorSeparate[img], {"LargestValues", 
                    5}], {"Centroid"}][[All, 2, 1]]][[1]] - 160)/
             200]] + Pi/2, {0, 0, 1}][{2, 0, 0}],
     Lighting -> {{"Directional", White, {0, 1, 0}}}
     ]
    ]
   ];

A shape:

ellipsoid = Scale[Sphere[{0, 0, 0}], {1, 1.5, 1}, {0, 0, 0}];

Let's take a look:

activeVision[ellipsoid]

Don't forget to quit your webcam when you're done:

IMAQ`StopCamera[]

Answer 2

Here is a prototype, mostly guessing at your intentions. This solution may cause motion sickness :)

Dynamic@DynamicModule[{image, binary, viewangle},
  binary = DeleteBorderComponents@Erosion[FillingTransform[
     ColorNegate@Binarize[image = CurrentImage[], FindThreshold[image]], 3], 10];
  viewangle = -\[Pi]/2 - (1 /. ComponentMeasurements[binary, "Orientation"]); 
  Graphics3D[{Opacity[0.5], Texture[SetAlphaChannel[image, ColorNegate@binary]], 
      Polygon[{{-3, 3, -2}, {3, 3, -2}, {3, 3, 4}, {-3, 3, 4}}, 
      VertexTextureCoordinates -> {{1, 0}, {0, 0}, {0, 1}, {1, 1}}], 
      Blue, Cylinder[], Green, Opacity[.3], 
      Cuboid[{-2, -2, -2}, {2, 2, -1}]}, 
      ViewVertical -> {Sin[viewangle], 0, Cos[viewangle]}]
]

A couple of example positions:

The image on the back plane is the current image capture, with the face mask showing what was used to obtain the orientation of my head using ComponentMeasurements. I used ViewVertical in this prototype. Extend as desired.

It is a rather surreal experience to see a graphic rotating in sync with your head movements in real time!

Answer 3

I have some experience using TrackIR, a system used for headtracking in flightsims etc. It operates using an IR camera and reflective dots on your face or cap. It works fast and reliable (as long as there aren't any blinding sources of IR light in your room). Using TIR2Joy and PPJoy (here and/or at GitHub) it can be used to emulate DirectX joysticks, and thus it can be read by Mathematica.

There are freeware replacements that do that too. FreeTrackNoir and Freetrack are related examples. They purportedly can emulate a DirectX 6DOF joystick using PPJoy.

I have reported some success in the past connecting Mathematica to Wii hardware. It should be not too difficult to use this for headtracking purposes, though unfortunately this involves wearing a Wiimote on your head, or alternatively, a few IR leds.