# How to segment rectangular photographs on a table?

Given an image of a table with photographs (take from any angle), I'm trying to extract the photographs and correctly align them into perfect rectangles. There are four steps to the algorithm:

1. Detect quadrangles in the input image

3. Re-project the quadrangles to a bird's-eye view

4. Correct for curved edges in the sub-images

Here's an example of doing the first two steps:

• For each of detected photographic quadrangles an orange outline is drawn.
• I've added a green one to emphasize the fidelity of solution I'm looking for, we can't lose any pixels in this process.
• Notice how the orange rectangles do not contain any white margins - we just want them to shrink-wrap the photographic content.

Ok, then here's what the remaining steps look like:

Note that for each of reprojected photograph (shown on the left) we fix the warping/curvature of the photograph by transforming it into a perfectly rectangular image (on the right).

Here's an example image to try:

Import @ CloudObject[
"https://www.wolframcloud.com/objects/d3999fcf-7a01-4d60-bb9c-38604d252475"]


And as requested, some additional examples to test with:

CloudGet @ CloudObject["https://www.wolframcloud.com/objects/068cf32f-b753-4c38-bc01-f378bc64d54a"]


• How to peel the labels from marmalade jars using Mathematica? is a possible duplicate. Aug 16 '18 at 23:23
• See ImagePerspectiveTransformation. Aug 16 '18 at 23:46
• @DavidG.Stork I think ImagePerspectiveTransformation will help to re-project the image but we would have to assume a fixed aspect (the most common print aspects are 5x7, 4x6, 8x10). But deriving correct segmentations in step 1 and 2 is the hardest part IMO
– M.R.
Aug 16 '18 at 23:49
• Aug 17 '18 at 5:50
• Your test image is a harder version of the problem than the first image in your post ;) The lighting is much harder, the background is not a uniform color, some photos are cut off by the border. Aug 24 '18 at 10:29

Here's an idea, but it only works (partially) with the easiest of the test cases.

img = CloudGet[CloudObject["https://www.wolframcloud.com/objects/068cf32f-b753-4c38-bc01-\
f378bc64d54a"]][[5]]


(* Binarize - this step depends a lot on the background in the image *)
img2 = ChanVeseBinarize[img, Automatic, {Automatic, Automatic, 2, 1}]


(* Find hulls, and discard ones that are tiny or touching a border *)
comps = MorphologicalComponents[img2, Method -> "ConvexHull"];
comps2 = SelectComponents[comps, #AdjacentBorderCount == 0 && #Count > 50 &];

(* show perimeters, to see how it's working *)
perims = ComponentMeasurements[comps2, "PerimeterPositions"];
HighlightImage[img, {PointSize[Medium], Red, Point @@@ perims[[All, 2]]}]


(* Extract the single images *)
singleimgs = ComponentMeasurements[ {comps2, img} , "MaskedImage"][[All, 2]]


transform[img_] :=
Module[{img2, frame, comp, p, c, c2, cornerdemo, t, img3},

(*get the perimeter of the image*)
frame = AlphaChannel[img2];
comp = MorphologicalComponents[frame, Method -> "ConvexHull"];
p = Flatten[
ComponentMeasurements[comp, "PerimeterPositions"][[1, 2]], 1];

(*find the corners of the image*)
c = Sort[ImageCorners[frame, 5, 0.01]];

(*the desired new corners*)
c2 = Sort[Tuples[MinMax /@ Transpose[p]]];

(*just to demonstrate the corner detection*)
colors = {Red, Orange, Yellow, Blue};
cornerdemo =
Show[frame,
Graphics[{Table[{colors[[n]], Disk[c2[[n]], 5]}, {n, 4}]}],
Graphics[Table[{colors[[n]], Circle[c[[n]], 5]}, {n, 4}]]];

(*transform such that the image corners are mapped to the desired corners*)
t = Last[FindGeometricTransform[c2, c]];
img3 = ImagePerspectiveTransformation[img2, t, DataRange -> Full];

{cornerdemo, img3}]

transform /@ singleimgs


With some of the examples given, we can bring all the images in the correct position at once, before extracting them. Here I chose 4 points using the coordinates tool.

imgs = CloudGet@
CloudObject[
"https://www.wolframcloud.com/objects/068cf32f-b753-4c38-bc01-\
f378bc64d54a"];

fourPoints = {{43.8398, 70.2148}, {336.398, 68.4375}, {342.199, 219.031}, {56.6094,209.16}}


This function processes the image. It takes the image and the four points.

processImage[img_, pts_List] := (
rect = {{pts[[1]][[1]], pts[[1]][[2]]}, {pts[[3]][[1]],
pts[[1]][[2]]}, {pts[[3]][[1]], pts[[3]][[2]]}, {pts[[1]][[1]],
pts[[3]][[2]]}};
transform = FindGeometricTransform[rect, pts][[2]];
Show[fullImage =
Sharpen[ImagePerspectiveTransformation[img, transform,
PlotRange -> Full], 1],
Graphics[{PointSize[Large], Green, Point[rect], Transparent,
EdgeForm[Green], Polygon[rect]}], ImageSize -> Medium]
)

processImage[imgs[[1]],fourPoints];


Now we isolate the pictures.

perim = MorphologicalPerimeter[fullImage, 60];
i2 = DeleteSmallComponents[perim];
morphBox = MorphologicalTransform[i2, {"BoundingBoxes", "Clean"}, Infinity];
ima=ImageMultiply[fullImage,morphBox];


Extract as a list of images and discard those not meeting a certain dimensions.

singles = ComponentMeasurements[ima, "MaskedImage"][[All, 2]];
validSingles =
Select[singles,
0.4 < Min[ImageDimensions[#]]/Max[ImageDimensions[#]] &]


Go through the list of images and attempt to trim the borders. It is difficult to get something that will work for all images, particularly when the border is the same color as the background of the photo. This works for some images, but not all of them.

imageList = {};
n = 1;
While[n <= Length[validSingles],
v = validSingles[[n]];
dim = ImageDimensions[v];
bd = BorderDimensions[ImageAdjust[v, {-0.1, 0.1}], 0.04];
crop1 = ImageCrop[v,
dim - {bd[[1]][[1]], bd[[2]][[1]]}, {Left, Bottom}];
dim2 = ImageDimensions[crop1];
crop2 = ImageCrop[crop1,
dim2 - {bd[[1]][[2]], bd[[2]][[2]]}, {Right, Top}];
AppendTo[imageList, crop2];
n++;
]
Row[imageList, "  "]


Better with a black background.