How to segment rectangular photographs on a table?

Question

Given an image of a table with photographs (take from any angle), I'm trying to extract the photographs and correctly align them into perfect rectangles. There are four steps to the algorithm:

1. Detect quadrangles in the input image

2. Filter out non-photograph quadrangles

3. Re-project the quadrangles to a bird's-eye view

4. Correct for curved edges in the sub-images

Here's an example of doing the first two steps:

• For each of detected photographic quadrangles an orange outline is drawn.
• I've added a green one to emphasize the fidelity of solution I'm looking for, we can't lose any pixels in this process.
• Notice how the orange rectangles do not contain any white margins - we just want them to shrink-wrap the photographic content.

Ok, then here's what the remaining steps look like:

Note that for each of reprojected photograph (shown on the left) we fix the warping/curvature of the photograph by transforming it into a perfectly rectangular image (on the right).

Here's an example image to try:

Import @ CloudObject[
 "https://www.wolframcloud.com/objects/d3999fcf-7a01-4d60-bb9c-38604d252475"]

And as requested, some additional examples to test with:

CloudGet @ CloudObject["https://www.wolframcloud.com/objects/068cf32f-b753-4c38-bc01-f378bc64d54a"]

Answer 1

Here's an idea, but it only works (partially) with the easiest of the test cases.

img = CloudGet[CloudObject["https://www.wolframcloud.com/objects/068cf32f-b753-4c38-bc01-\
f378bc64d54a"]][[5]]

(* Binarize - this step depends a lot on the background in the image *)
img2 = ChanVeseBinarize[img, Automatic, {Automatic, Automatic, 2, 1}]

(* Find hulls, and discard ones that are tiny or touching a border *)
comps = MorphologicalComponents[img2, Method -> "ConvexHull"];
comps2 = SelectComponents[comps, #AdjacentBorderCount == 0 && #Count > 50 &];

(* show perimeters, to see how it's working *)
perims = ComponentMeasurements[comps2, "PerimeterPositions"];
HighlightImage[img, {PointSize[Medium], Red, Point @@@ perims[[All, 2]]}]

(* Extract the single images *)
singleimgs = ComponentMeasurements[ {comps2, img} , "MaskedImage"][[All, 2]]

transform[img_] := 
 Module[{img2, frame, comp, p, c, c2, cornerdemo, t, img3},
  
  (*add some padding,to make the corner detection work better*)
  img2 = ImagePad[img, 2];
  
  (*get the perimeter of the image*)
  frame = AlphaChannel[img2];
  comp = MorphologicalComponents[frame, Method -> "ConvexHull"];
  p = Flatten[
    ComponentMeasurements[comp, "PerimeterPositions"][[1, 2]], 1];
  
  (*find the corners of the image*)
  c = Sort[ImageCorners[frame, 5, 0.01]];
  
  (*the desired new corners*)
  c2 = Sort[Tuples[MinMax /@ Transpose[p]]];
  
  (*just to demonstrate the corner detection*)
  colors = {Red, Orange, Yellow, Blue};
  cornerdemo = 
   Show[frame, 
    Graphics[{Table[{colors[[n]], Disk[c2[[n]], 5]}, {n, 4}]}], 
    Graphics[Table[{colors[[n]], Circle[c[[n]], 5]}, {n, 4}]]];
  
  (*transform such that the image corners are mapped to the desired corners*)
  t = Last[FindGeometricTransform[c2, c]];
  img3 = ImagePerspectiveTransformation[img2, t, DataRange -> Full];
  
  {cornerdemo, img3}]

transform /@ singleimgs

Answer 2

With some of the examples given, we can bring all the images in the correct position at once, before extracting them. Here I chose 4 points using the coordinates tool.

imgs = CloudGet@
   CloudObject[
    "https://www.wolframcloud.com/objects/068cf32f-b753-4c38-bc01-\
f378bc64d54a"];

fourPoints = {{43.8398, 70.2148}, {336.398, 68.4375}, {342.199, 219.031}, {56.6094,209.16}}

This function processes the image. It takes the image and the four points.

processImage[img_, pts_List] := (
  rect = {{pts[[1]][[1]], pts[[1]][[2]]}, {pts[[3]][[1]], 
     pts[[1]][[2]]}, {pts[[3]][[1]], pts[[3]][[2]]}, {pts[[1]][[1]], 
     pts[[3]][[2]]}};
  transform = FindGeometricTransform[rect, pts][[2]];
  Show[fullImage = 
    Sharpen[ImagePerspectiveTransformation[img, transform, 
      PlotRange -> Full], 1], 
   Graphics[{PointSize[Large], Green, Point[rect], Transparent, 
     EdgeForm[Green], Polygon[rect]}], ImageSize -> Medium]
  )

processImage[imgs[[1]],fourPoints];

Now we isolate the pictures.

perim = MorphologicalPerimeter[fullImage, 60];
i2 = DeleteSmallComponents[perim];
morphBox = MorphologicalTransform[i2, {"BoundingBoxes", "Clean"}, Infinity]; 
ima=ImageMultiply[fullImage,morphBox];

Extract as a list of images and discard those not meeting a certain dimensions.

singles = ComponentMeasurements[ima, "MaskedImage"][[All, 2]];
validSingles = 
 Select[singles, 
  0.4 < Min[ImageDimensions[#]]/Max[ImageDimensions[#]] &]

Go through the list of images and attempt to trim the borders. It is difficult to get something that will work for all images, particularly when the border is the same color as the background of the photo. This works for some images, but not all of them.

imageList = {};
n = 1;
While[n <= Length[validSingles],
 v = validSingles[[n]];
 dim = ImageDimensions[v];
 bd = BorderDimensions[ImageAdjust[v, {-0.1, 0.1}], 0.04];
 crop1 = ImageCrop[v, 
   dim - {bd[[1]][[1]], bd[[2]][[1]]}, {Left, Bottom}];
 dim2 = ImageDimensions[crop1];
 crop2 = ImageCrop[crop1, 
   dim2 - {bd[[1]][[2]], bd[[2]][[2]]}, {Right, Top}];
 AppendTo[imageList, crop2];
 n++;
 ]
Row[imageList, "  "]

Better with a black background.