How can I set the view vector in a 3D scene by tracking my face's position?

Faking 3D with 2D tracking data is something that I've long wanted to implement for use with my front facing laptop camera. A first step to solving this would be dynamically setting the viewpoint according to the movement of your head.

Edit:

To clarify, I'm looking for an answer like the code I'm giving below. The reason I ask is because there were a number of problems with my own solution:

1. The method used is not sensitive enough
2. It is far too slow
3. Not robust to background movement

So a real answer to this question might use some fancy bit of machine vision, or interface with opencv face tracking libraries, or something else entirely. Here is a video of this technique applied sucessfully: http://www.youtube.com/watch?v=7saRtCUgbNI

ClusterMean[x_, 0, _] := Mean[x];
ClusterMean[{x_}, n_, frac_] := x;
ClusterMean[x_, n_, frac_] :=
With[{m = Mean[x], sd = StandardDeviation[x]},
With[{x2 = Select[x, Norm[Norm[# - m]/sd] < frac &]},
If[x2 == {}, Mean[x], ClusterMean[x2, n - 1, frac]]]];
$ytrim =$xtrim = {1, All};
SmallCurrentImage[] := ImageTake[CurrentImage[], $ytrim,$xtrim];

$first = SmallCurrentImage[]; FaceOffset[center : {_, _}] := Module[{mean, img1, img2, matches}, mean = {0, 0}; img1 =$first;
$currentimage = img2 = SmallCurrentImage[];$matches = matches = ImageCorrespondingPoints[img1, img2];

If[mean == {}, center, ClusterMean[mean, 1, 0.8]]
]

center = {0, 0}; center2 = {0, 0}; $first = SmallCurrentImage[]; Dynamic[ If[CurrentValue["MouseOver"], center = FaceOffset[center]; center2 = center2*0.5 + center*0.5;]; Refresh[ Labeled[ Refresh[ Graphics3D[{PolyhedronData["RhombicHexecontahedron", "Faces"]}, ViewPoint -> Refresh[5*Normalize[Prepend[-center2/80, -1]], TrackedSymbols :> {center}], SphericalRegion -> True, RotationAction -> "Clip"], None], {Show[$currentimage,
Graphics[{Red, MapThread[Line[{#1, #2}] &, $matches]}]], Button["Reset View", ($first = SmallCurrentImage[];
center2 = {0, 0};), Appearance -> "Palette",
ImageSize -> Automatic]}, {Left, {Bottom, Right}}],
TrackedSymbols :> {center}]]

-
That's not a whole lot go by. Are you looking for an outline of a method which you will then implement yourself? Is there a specific part of the process that you need help with? Do you have any experience with motion capture? What have you already tried? –  Mr.Wizard Jun 8 '12 at 4:12
Please read the FAQ on admissible questions. Ask questions on problems you actually face and take care they are reasonably scoped. –  Sjoerd C. de Vries Jun 8 '12 at 5:31
Ok, I rephrased the question and gave example code to sketch the idea... –  M.R. Jun 8 '12 at 7:32
Is there anyway to make this work under Linux? –  image_doctor Jun 8 '12 at 8:59
@image_doctor I'm not sure about that. –  M.R. Jun 8 '12 at 13:30

You might try to expand the example below for a more generic solution. This function uses the webcam to track a red object and updates the viewpoint according to the position of this object. You draw a nice red bindi on your forehead, move the head left or right and the viewpoint will update as a function of your head position. Right now it works for left/right translations only.

  activeVision[shape_] :=
Dynamic[
With[{img = CurrentImage[]},
Graphics3D[
shape,
SphericalRegion -> True, Boxed -> False, AspectRatio -> 1,
Background -> Black, ViewPoint -> RotationTransform[-N[ArcTan[
(Mean[
ComponentMeasurements[
Threshold[
ImageSubtract[ImageSubtract[#1, #2], #3] &  @@
ColorSeparate[img], {"LargestValues",
5}], {"Centroid"}][[All, 2, 1]]][[1]] - 160)/
200]] + Pi/2, {0, 0, 1}][{2, 0, 0}],
Lighting -> {{"Directional", White, {0, 1, 0}}}
]
]
];


A shape:

ellipsoid = Scale[Sphere[{0, 0, 0}], {1, 1.5, 1}, {0, 0, 0}];


Let's take a look:

activeVision[ellipsoid]


Don't forget to quit your webcam when you're done:

IMAQStopCamera[]

-
Nice, but having to draw on yourself is a drawback I think. –  M.R. Jun 8 '12 at 8:33
@Mike It is actually able to track your lips, but it depends on the lighting conditions and on the color of the lips relative to the color of the face. To improve performance you might use a nice red lipstick... –  VLC Jun 8 '12 at 8:40
lipstick, check! –  M.R. Jun 8 '12 at 9:09

I have some experience using TrackIR, a system used for headtracking in flightsims etc. It operates using an IR camera and reflective dots on your face or cap. It works fast and reliable (as long as there aren't any blinding sources of IR light in your room). Using TIR2Joy and PPJoy (here and/or at GitHub) it can be used to emulate DirectX joysticks, and thus it can be read by Mathematica.

There are freeware replacements that do that too. FreeTrackNoir and Freetrack are related examples. They purportedly can emulate a DirectX 6DOF joystick using PPJoy.

I have reported some success in the past connecting Mathematica to Wii hardware. It should be not too difficult to use this for headtracking purposes, though unfortunately this involves wearing a Wiimote on your head, or alternatively, a few IR leds.

-

Here is a prototype, mostly guessing at your intentions. This solution may cause motion sickness :)

Dynamic@DynamicModule[{image, binary, viewangle},
binary = DeleteBorderComponents@Erosion[FillingTransform[
ColorNegate@Binarize[image = CurrentImage[], FindThreshold[image]], 3], 10];
viewangle = -\[Pi]/2 - (1 /. ComponentMeasurements[binary, "Orientation"]);
Graphics3D[{Opacity[0.5], Texture[SetAlphaChannel[image, ColorNegate@binary]],
Polygon[{{-3, 3, -2}, {3, 3, -2}, {3, 3, 4}, {-3, 3, 4}},
VertexTextureCoordinates -> {{1, 0}, {0, 0}, {0, 1}, {1, 1}}],
Blue, Cylinder[], Green, Opacity[.3],
Cuboid[{-2, -2, -2}, {2, 2, -1}]},
ViewVertical -> {Sin[viewangle], 0, Cos[viewangle]}]
]


A couple of example positions:

The image on the back plane is the current image capture, with the face mask showing what was used to obtain the orientation of my head using ComponentMeasurements. I used ViewVertical` in this prototype. Extend as desired.

It is a rather surreal experience to see a graphic rotating in sync with your head movements in real time!

-
Surreal it is! I'm sure one could make some cool interactive 3d modern art out of this. –  M.R. Jun 8 '12 at 9:07
Don't forget to turn off the camera! :) –  rcollyer Jun 8 '12 at 14:22