2
$\begingroup$

I have two images:

img1image 1

and image 2 enter image description here

I want to replace the eyes of image 1 with those of image 2. I want to do this in an automatic fashion because I want to try this with several images of this face that are not well aligned but all have this front view perspective.

My first approach was to use FacialFeatures[img,{"RightEyePoints", "LeftEyePoints"}] in both pictures to detect the eyes. This works reasonably well. The eyes are roughly detected (see image 2 as an example):

enter image description here

at this point, I'm kind of stuck and do not know how to proceed further. I have to somehow cut out the eyes and replace them in the pictures, but this is difficult since the eyepoints do not circumscribe the eyes well.

I was hoping for some inputs on how something like this is "normally" done or can be alternatively done in an easier way. I have no experience so far, but I think this should be fairly easy for somebody with experience.

$\endgroup$
8
  • $\begingroup$ I could give it a try later, but I think it would help with a couple of more images. Ideally they should represent the maximum distortion that will be present (the largest amount of head movement in the series of images that you want to do this on). $\endgroup$
    – C. E.
    Oct 5 '20 at 11:59
  • 1
    $\begingroup$ The eyes may be difficult to detect because they can be opened/closed etc. but there are other features that are quite distinguishable. For example, let's say you crop out the nostrils or (part of) an eyebrow, or an ear, then if the head movements are not too large I would expect to be able to find these in the images using the ImageCorrelate function (see the applications in the documentation for a similar example). Let's say you can find four features like this (each nostril can be a feature, perhaps) then you have these four points that you know... $\endgroup$
    – C. E.
    Oct 5 '20 at 12:55
  • 1
    $\begingroup$ ...and you also know these four points in the original image that you cropped from. You can use FindGeometricTransform to find a transform from the points you found to the points in the original image, and vice-versa. You can now take points on the eyes in the original image and transform them to the new image where those points are unknown. And there you have it, eye points even though you never had to look for the eyes themselves. $\endgroup$
    – C. E.
    Oct 5 '20 at 12:57
  • 1
    $\begingroup$ This works only under some assumptions, noticeably that the feature points are coplanar. However, if the head movement is small enough I think it will give a decent result, especially if you can use the eyebrows and nostrils because those are roughly coplanar with the eyes. $\endgroup$
    – C. E.
    Oct 5 '20 at 13:00
  • 1
    $\begingroup$ I didn’t have time to extend my answer, but another related approach you could try is to extract the eyes as templates as in my answer. Then you will directly detect them. You can use multiple templates with open, closed, and half-closed eyelids and pick the one that produces the best correlation. $\endgroup$
    – C. E.
    Oct 5 '20 at 22:19
3
$\begingroup$

ok, here goes my idea. Even if it doesn't pan out perfectly it might serve as an introduction to image alignment (aka image registration).

Basic definitions

img = ColorConvert[Import["https://i.stack.imgur.com/eFjUe.png"], "Grayscale"];
img2 = ColorConvert[Import["https://i.stack.imgur.com/pnTX6.png"], "Grayscale"];

highlight[img_, pts_] := HighlightImage[img, {PointSize[Large], Red, Point[pts]}]

We extract the positions of the eye corners in img:

eyeCorners = {{1052.58, 881.74}, {1208.43, 879.34}, {1400.24, 886.54}, {1568.08, 900.92}};
highlight[img, eyeCorners]

Eye corners

What follows is two different methods to transfer these eye corner points onto the second image.

Extracting features manually

For the method that I will demonstrate to work I need to detect several corresponding points in the images. It doesn't matter which points they are, as long as they match. I pick some points that have disinguishable surroundings:

pts = {{987.84, 987.63}, {1604.05, 1016.41}, {1251.58, 
    649.57}, {1364.28, 656.76}, {1822.23, 867.75}};
rois = Rectangle[# - {50, 50}, # + {50, 50}] & /@ pts;
templates = ImageTrim[img, rois];
templates

Templates

I can use the templates to find the corresponding points in the second image as follows.

findLandmarks[img_, templates_] := Module[{correlation},
  correlation = Map[
    ImageCorrelate[img, #, NormalizedSquaredEuclideanDistance] &,
    templates
    ];
  First@PixelValuePositions[#, "Min"] & /@ correlation
  ]

landmarks2 = findLandmarks[img2, templates];
highlight[img2, landmarks2]

Landmarks

As you can see, the templates from the first image allow me to find the corresponding points in the second image, using image correlation.

Now the plan is to find a function that transforms points in the first image to points in the second image.

{err, tf} = FindGeometricTransform[landmarks2, pts];
highlight[img2, tf@pts]

Transformed landmarks

The magic of the transformation function is that it allows us to take points in the first image and plot points in the second image, which is what we want to do with the eye corners.

highlight[img2, tf@eyeCorners]

Transformed eye corners

Alas, it does not work out that well. The precision is not good enough. Or maybe it is partly a visual effect of the eye closing that what looks like the eye corner looks like it is part of the eyelid?

Mathematically, this method is actually restricted to coplanar corresponding points. It works great if you're doing it with images of a game board for example, but less great when you're doing it with 3D surfaces like a face. However, in this case I think it looks like the movement is just simple translation, and therefore I think it should work, especially since the landmarks are roughly coplanar.

We now move on to a better, automated version of this approach.

Extracting features automatically

With the previous approach, we manually picked some landmarks that we felt were distinguishable. However, there algorithms that will pick out distinguishable points themselves. Then, having found distinguishable points in two images, it will match them, and we end up with corresponding points without having to select any landmarks ourselves. It is very simple to implement in Mathematica, it looks like this:

{pts1, pts2} = ImageCorrespondingPoints[img, img2];
{err, tf} = FindGeometricTransform[pts2, pts1];
highlight[img2, tf@eyeCorners]

Transformed eye corners.

The code is simpler but the result looks quite bad.

Further reading

If you want to know more about how ImageCorrespondingPoints works, you should check out the ImageKeypoints function. ImageCorrespondingPoints basically runs ImageKeypoints on both of the images, then takes the "keypoint descriptors" that it finds, and matches the keypoint descriptors that are closest to each other. (How distance is measured varies depending on what type of descriptor it is, but it can be as simple as a Euclidean distance).

If we think of how FindGeometricTransform was used here, we can see that we can basically implement ImageAlign ourselves at this point, so that function might also be interesting to read about.

If you found this interesting and want to know more about FindGeometricTransform, then you might look up the "DLT algorithm". (Deep discussions of this deceptively simple answer are found in books on so-called multiple view geometry.)

$\endgroup$
1
  • $\begingroup$ Thank you very much ;)! $\endgroup$
    – holistic
    Oct 6 '20 at 9:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.