# Video stabilization: extracting the transformation rules used by ImageAlign

Imagine that I have a real life video, with all the artefacts coming from real life, and also from video compression, etc, and that I can edit this video, with filters and cropping, in such a way that I can improve the success of ImageAlign as an image stabilizer. How can I align/stabilize the original video, using the transformation rules identified for the edited one?

(I think that) This is not a question on how to recreate the ImageAlign function, from other functions belonging to the same family (FindGeometricTransform, etc).

(I'm sure that) this is not a question on what Options would make it work better on the specific example I putted together (unless there's an option that gets these transformation rules out..., and that I completely missed)

Let's start by creating a shaky film:

film = Rasterize /@ Table[
Graphics3D[{EdgeForm[None],
Blue,
GeometricTransformation[Cuboid[{-4, -4, 0}],
Table[RotationTransform[
i*Pi/2 + angle, {0, 0, 1}, {0, 0, 0}], {i, 4}]],
Red,
GeometricTransformation[Cuboid[{-4, -4, 0}],
Table[RotationTransform[i*Pi/2, {0, 0, 1}, {0, 0, 0}], {i,
4}]]},
Boxed -> False,
ViewVector -> {5 {Pi, Pi/2, 2}, {0 + RandomReal[],
0 + RandomReal[], 0}}, ViewAngle -> Pi/5],
{angle, 0, 2 Pi/4 - Pi/32, Pi/32}
];

ListAnimate[film]


And now, let's try to unshake it:

filmSteady1 = ImageAlign[film[[1]], #, TransformationClass -> "Rigid"] & /@ film;


(First side note: if we do not Rasterize it before, interestingly, it still works, but there's probably a bug in the internals of the ImageAlign function, or in compile, since I keep receiving the message: "CompiledFunction::cfn: Numerical error encountered at instruction 8; proceeding with uncompiled evaluation. >>"). I'm not expert enough to understand its origins. Can someone try to catch this one, so that it gets reported?

ListAnimate[filmSteady1]


Now, lets try ImageAlign with a very simple filter added (again, this is an example; one can think of masks, two completely different videos, coming from two different recording technologies, etc):

infoFilm = ColorReplace[#, Blue -> White, 0.3] & /@ film;
filmSteady2 = ImageAlign[infoFilm[[1]], #, TransformationClass -> "Rigid"] & /@ infoFilm;



Much better! How can I then apply to the target/original video, the identified transformation rules?

Second side note: if instead of using "Blue->White", you just write "Blue", the results from ImageAlign are not the same... Shouldn't there be a way for ImageAlign to keep the transparent information? (I imagine this is coming from the background option, but it makes more sense on image borders, than in the middle of the image).

Third side note: Once the transformation rules are listed, it is possible to imagine a multitude of techniques that would probably be more adapted to a video: comparing each image/frame to the previous one (in time), and then, pre-combining the transformation rules all together, before processing the image; analyze just every n frames, and analytically determine halfway transformations to apply to the non analyzed frames (so to go faster); determine a smooth transformation "path", obeying to some spline rule or similar; etc...

• I imagine that ImageAlign uses ImageKeypoints and FindGeometricTransform internally. Can't you do the same, in order to get access to the transformation? May 2, 2015 at 13:25
• @OleksandrR. the options seem to indicate that ImageAlign has extra tricks available. May 2, 2015 at 13:30
• Really? Maybe in version 10. I only have version 9 and the options there are the same. Would be worth trying the ImageAlign options as undocumented possibilities for FindGeometricTransform, perhaps. May 2, 2015 at 13:32
• @OleksandrR. Version 10 introduced as methods: Keypoints, MeanSquareGradientDescent, Fourier, FourierBlurInvariant. Indeed, the FindGeometricTrasnformation has {"ImageAlign",method}, which should do the trick. I will try it, and post the answer. Jun 4, 2015 at 12:06

Following the comments from Oleksandr R.:

aux = Import["input.gif"];
i1 = ColorReplace[aux[[1]], Blue -> White, 0.3];
ImageTransformation[#,
FindGeometricTransform[ColorReplace[#, Blue -> White, 0.3], i1,
TransformationClass -> "Rigid", Method -> {"ImageAlign"}][[2]],
DataRange -> Full, Background -> White, Masking -> All] & /@ aux


Lets see if the result is exactly the same (with ImageSubtract):

A few remarks are in order:

• the result is not 100% the same. should this be reported? (is the TransformationClass being passed to the ImageAlign method?)
• you will notice that I used the FindGeometricTransform with the two first arguments inverted in respect to (what is my reading from) the documentation: "The function trfun can be applied to the positions pts2 to align them with the positions pts1". Is this a bug?
• is it possible to completely remove the black line at the image limits (why doesn't Background work to the full extents? Is there a better way of doing it, or would this be an "improvement" request?