Faking 3D with 2D tracking data is something that I've long wanted to implement for use with my front facing laptop camera. A first step to solving this would be dynamically setting the viewpoint according to the movement of your head.


To clarify, I'm looking for an answer like the code I'm giving below. The reason I ask is because there were a number of problems with my own solution:

  1. The method used is not sensitive enough
  2. It is far too slow
  3. Not robust to background movement

So a real answer to this question might use some fancy bit of machine vision, or interface with opencv face tracking libraries, or something else entirely. Here is a video of this technique applied sucessfully: http://www.youtube.com/watch?v=7saRtCUgbNI

ClusterMean[x_, 0, _] := Mean[x];
ClusterMean[{x_}, n_, frac_] := x;
ClusterMean[x_, n_, frac_] := 
  With[{m = Mean[x], sd = StandardDeviation[x]}, 
   With[{x2 = Select[x, Norm[Norm[# - m]/sd] < frac &]}, 
    If[x2 == {}, Mean[x], ClusterMean[x2, n - 1, frac]]]];
$ytrim = $xtrim = {1, All};
SmallCurrentImage[] := ImageTake[CurrentImage[], $ytrim, $xtrim];

$first = SmallCurrentImage[];

FaceOffset[center : {_, _}] :=

 Module[{mean, img1, img2, matches}, mean = {0, 0};
  img1 = $first;
      $currentimage = img2 = SmallCurrentImage[];
  $matches = matches = ImageCorrespondingPoints[img1, img2];
  mean = MapThread[Subtract, matches];

  If[mean == {}, center, ClusterMean[mean, 1, 0.8]]

center = {0, 0}; center2 = {0, 0}; $first = SmallCurrentImage[];

 If[CurrentValue["MouseOver"], center = FaceOffset[center];
  center2 = center2*0.5 + center*0.5;];

     ViewPoint -> 
      Refresh[5*Normalize[Prepend[-center2/80, -1]], 
       TrackedSymbols :> {center}], SphericalRegion -> True, 
     RotationAction -> "Clip"], 
    None], {Show[$currentimage, 
         Graphics[{Red, MapThread[Line[{#1, #2}] &, $matches]}]], 
        Button["Reset View", ($first = SmallCurrentImage[]; 
      center2 = {0, 0};), Appearance -> "Palette", 
     ImageSize -> Automatic]}, {Left, {Bottom, Right}}], 
  TrackedSymbols :> {center}]]
  • 1
    $\begingroup$ That's not a whole lot go by. Are you looking for an outline of a method which you will then implement yourself? Is there a specific part of the process that you need help with? Do you have any experience with motion capture? What have you already tried? $\endgroup$
    – Mr.Wizard
    Jun 8, 2012 at 4:12
  • 1
    $\begingroup$ Please read the FAQ on admissible questions. Ask questions on problems you actually face and take care they are reasonably scoped. $\endgroup$ Jun 8, 2012 at 5:31
  • 1
    $\begingroup$ Ok, I rephrased the question and gave example code to sketch the idea... $\endgroup$
    – M.R.
    Jun 8, 2012 at 7:32
  • $\begingroup$ Is there anyway to make this work under Linux? $\endgroup$ Jun 8, 2012 at 8:59
  • $\begingroup$ @image_doctor I'm not sure about that. $\endgroup$
    – M.R.
    Jun 8, 2012 at 13:30

3 Answers 3


You might try to expand the example below for a more generic solution. This function uses the webcam to track a red object and updates the viewpoint according to the position of this object. You draw a nice red bindi on your forehead, move the head left or right and the viewpoint will update as a function of your head position. Right now it works for left/right translations only.

  activeVision[shape_] :=
   With[{img = CurrentImage[]},
     SphericalRegion -> True, Boxed -> False, AspectRatio -> 1, 
     Background -> Black, ViewPoint -> RotationTransform[-N[ArcTan[
                    ImageSubtract[ImageSubtract[#1, #2], #3] &  @@ 
                    ColorSeparate[img], {"LargestValues", 
                    5}], {"Centroid"}][[All, 2, 1]]][[1]] - 160)/
             200]] + Pi/2, {0, 0, 1}][{2, 0, 0}],
     Lighting -> {{"Directional", White, {0, 1, 0}}}

A shape:

ellipsoid = Scale[Sphere[{0, 0, 0}], {1, 1.5, 1}, {0, 0, 0}];

Let's take a look:


Don't forget to quit your webcam when you're done:

  • $\begingroup$ Nice, but having to draw on yourself is a drawback I think. $\endgroup$
    – M.R.
    Jun 8, 2012 at 8:33
  • 6
    $\begingroup$ @Mike It is actually able to track your lips, but it depends on the lighting conditions and on the color of the lips relative to the color of the face. To improve performance you might use a nice red lipstick... $\endgroup$
    – VLC
    Jun 8, 2012 at 8:40
  • $\begingroup$ lipstick, check! $\endgroup$
    – M.R.
    Jun 8, 2012 at 9:09

Here is a prototype, mostly guessing at your intentions. This solution may cause motion sickness :)

Dynamic@DynamicModule[{image, binary, viewangle},
  binary = DeleteBorderComponents@Erosion[FillingTransform[
     ColorNegate@Binarize[image = CurrentImage[], FindThreshold[image]], 3], 10];
  viewangle = -\[Pi]/2 - (1 /. ComponentMeasurements[binary, "Orientation"]); 
  Graphics3D[{Opacity[0.5], Texture[SetAlphaChannel[image, ColorNegate@binary]], 
      Polygon[{{-3, 3, -2}, {3, 3, -2}, {3, 3, 4}, {-3, 3, 4}}, 
      VertexTextureCoordinates -> {{1, 0}, {0, 0}, {0, 1}, {1, 1}}], 
      Blue, Cylinder[], Green, Opacity[.3], 
      Cuboid[{-2, -2, -2}, {2, 2, -1}]}, 
      ViewVertical -> {Sin[viewangle], 0, Cos[viewangle]}]

A couple of example positions:

Head to the right

Head to the left

The image on the back plane is the current image capture, with the face mask showing what was used to obtain the orientation of my head using ComponentMeasurements. I used ViewVertical in this prototype. Extend as desired.

It is a rather surreal experience to see a graphic rotating in sync with your head movements in real time!

  • $\begingroup$ Surreal it is! I'm sure one could make some cool interactive 3d modern art out of this. $\endgroup$
    – M.R.
    Jun 8, 2012 at 9:07
  • 1
    $\begingroup$ Don't forget to turn off the camera! :) $\endgroup$
    – rcollyer
    Jun 8, 2012 at 14:22

I have some experience using TrackIR, a system used for headtracking in flightsims etc. It operates using an IR camera and reflective dots on your face or cap. It works fast and reliable (as long as there aren't any blinding sources of IR light in your room). Using TIR2Joy and PPJoy (here and/or at GitHub) it can be used to emulate DirectX joysticks, and thus it can be read by Mathematica.

There are freeware replacements that do that too. FreeTrackNoir and Freetrack are related examples. They purportedly can emulate a DirectX 6DOF joystick using PPJoy.

I have reported some success in the past connecting Mathematica to Wii hardware. It should be not too difficult to use this for headtracking purposes, though unfortunately this involves wearing a Wiimote on your head, or alternatively, a few IR leds.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.