Replace eyes in pictures

Question

I have two images:

img1

and image 2

I want to replace the eyes of image 1 with those of image 2. I want to do this in an automatic fashion because I want to try this with several images of this face that are not well aligned but all have this front view perspective.

My first approach was to use FacialFeatures[img,{"RightEyePoints", "LeftEyePoints"}] in both pictures to detect the eyes. This works reasonably well. The eyes are roughly detected (see image 2 as an example):

at this point, I'm kind of stuck and do not know how to proceed further. I have to somehow cut out the eyes and replace them in the pictures, but this is difficult since the eyepoints do not circumscribe the eyes well.

I was hoping for some inputs on how something like this is "normally" done or can be alternatively done in an easier way. I have no experience so far, but I think this should be fairly easy for somebody with experience.

Answer 1

ok, here goes my idea. Even if it doesn't pan out perfectly it might serve as an introduction to image alignment (aka image registration).

Basic definitions

img = ColorConvert[Import["https://i.stack.imgur.com/eFjUe.png"], "Grayscale"];
img2 = ColorConvert[Import["https://i.stack.imgur.com/pnTX6.png"], "Grayscale"];

highlight[img_, pts_] := HighlightImage[img, {PointSize[Large], Red, Point[pts]}]

We extract the positions of the eye corners in img:

eyeCorners = {{1052.58, 881.74}, {1208.43, 879.34}, {1400.24, 886.54}, {1568.08, 900.92}};
highlight[img, eyeCorners]

What follows is two different methods to transfer these eye corner points onto the second image.

Extracting features manually

For the method that I will demonstrate to work I need to detect several corresponding points in the images. It doesn't matter which points they are, as long as they match. I pick some points that have disinguishable surroundings:

pts = {{987.84, 987.63}, {1604.05, 1016.41}, {1251.58, 
    649.57}, {1364.28, 656.76}, {1822.23, 867.75}};
rois = Rectangle[# - {50, 50}, # + {50, 50}] & /@ pts;
templates = ImageTrim[img, rois];
templates

I can use the templates to find the corresponding points in the second image as follows.

findLandmarks[img_, templates_] := Module[{correlation},
  correlation = Map[
    ImageCorrelate[img, #, NormalizedSquaredEuclideanDistance] &,
    templates
    ];
  First@PixelValuePositions[#, "Min"] & /@ correlation
  ]

landmarks2 = findLandmarks[img2, templates];
highlight[img2, landmarks2]

As you can see, the templates from the first image allow me to find the corresponding points in the second image, using image correlation.

Now the plan is to find a function that transforms points in the first image to points in the second image.

{err, tf} = FindGeometricTransform[landmarks2, pts];
highlight[img2, tf@pts]

The magic of the transformation function is that it allows us to take points in the first image and plot points in the second image, which is what we want to do with the eye corners.

highlight[img2, tf@eyeCorners]

Alas, it does not work out that well. The precision is not good enough. Or maybe it is partly a visual effect of the eye closing that what looks like the eye corner looks like it is part of the eyelid?

Mathematically, this method is actually restricted to coplanar corresponding points. It works great if you're doing it with images of a game board for example, but less great when you're doing it with 3D surfaces like a face. However, in this case I think it looks like the movement is just simple translation, and therefore I think it should work, especially since the landmarks are roughly coplanar.

We now move on to a better, automated version of this approach.

Extracting features automatically

With the previous approach, we manually picked some landmarks that we felt were distinguishable. However, there algorithms that will pick out distinguishable points themselves. Then, having found distinguishable points in two images, it will match them, and we end up with corresponding points without having to select any landmarks ourselves. It is very simple to implement in Mathematica, it looks like this:

{pts1, pts2} = ImageCorrespondingPoints[img, img2];
{err, tf} = FindGeometricTransform[pts2, pts1];
highlight[img2, tf@eyeCorners]

The code is simpler but the result looks quite bad.

Further reading

If you want to know more about how ImageCorrespondingPoints works, you should check out the ImageKeypoints function. ImageCorrespondingPoints basically runs ImageKeypoints on both of the images, then takes the "keypoint descriptors" that it finds, and matches the keypoint descriptors that are closest to each other. (How distance is measured varies depending on what type of descriptor it is, but it can be as simple as a Euclidean distance).

If we think of how FindGeometricTransform was used here, we can see that we can basically implement ImageAlign ourselves at this point, so that function might also be interesting to read about.

If you found this interesting and want to know more about FindGeometricTransform, then you might look up the "DLT algorithm". (Deep discussions of this deceptively simple answer are found in books on so-called multiple view geometry.)