15
$\begingroup$

Given an image of a table with photographs (take from any angle), I'm trying to extract the photographs and correctly align them into perfect rectangles. There are four steps to the algorithm:

  1. Detect quadrangles in the input image

  2. Filter out non-photograph quadrangles

  3. Re-project the quadrangles to a bird's-eye view

  4. Correct for curved edges in the sub-images

Here's an example of doing the first two steps:

enter image description here

  • For each of detected photographic quadrangles an orange outline is drawn.
  • I've added a green one to emphasize the fidelity of solution I'm looking for, we can't lose any pixels in this process.
  • Notice how the orange rectangles do not contain any white margins - we just want them to shrink-wrap the photographic content.

Ok, then here's what the remaining steps look like:

enter image description here

Note that for each of reprojected photograph (shown on the left) we fix the warping/curvature of the photograph by transforming it into a perfectly rectangular image (on the right).

Here's an example image to try:

Import @ CloudObject[
 "https://www.wolframcloud.com/objects/d3999fcf-7a01-4d60-bb9c-38604d252475"]

And as requested, some additional examples to test with:

CloudGet @ CloudObject["https://www.wolframcloud.com/objects/068cf32f-b753-4c38-bc01-f378bc64d54a"]

enter image description here

$\endgroup$
14
  • 1
    $\begingroup$ How to peel the labels from marmalade jars using Mathematica? is a possible duplicate. $\endgroup$
    – C. E.
    Aug 16 '18 at 23:23
  • 2
    $\begingroup$ See ImagePerspectiveTransformation. $\endgroup$ Aug 16 '18 at 23:46
  • 1
    $\begingroup$ @DavidG.Stork I think ImagePerspectiveTransformation will help to re-project the image but we would have to assume a fixed aspect (the most common print aspects are 5x7, 4x6, 8x10). But deriving correct segmentations in step 1 and 2 is the hardest part IMO $\endgroup$
    – M.R.
    Aug 16 '18 at 23:49
  • 1
    $\begingroup$ Related: mathematica.stackexchange.com/q/126135/242 $\endgroup$ Aug 17 '18 at 5:50
  • 2
    $\begingroup$ Your test image is a harder version of the problem than the first image in your post ;) The lighting is much harder, the background is not a uniform color, some photos are cut off by the border. $\endgroup$
    – Carl Lange
    Aug 24 '18 at 10:29
8
+200
$\begingroup$

Here's an idea, but it only works (partially) with the easiest of the test cases.

img = CloudGet[CloudObject["https://www.wolframcloud.com/objects/068cf32f-b753-4c38-bc01-\
f378bc64d54a"]][[5]]

enter image description here

(* Binarize - this step depends a lot on the background in the image *)
img2 = ChanVeseBinarize[img, Automatic, {Automatic, Automatic, 2, 1}]

enter image description here

(* Find hulls, and discard ones that are tiny or touching a border *)
comps = MorphologicalComponents[img2, Method -> "ConvexHull"];
comps2 = SelectComponents[comps, #AdjacentBorderCount == 0 && #Count > 50 &];

(* show perimeters, to see how it's working *)
perims = ComponentMeasurements[comps2, "PerimeterPositions"];
HighlightImage[img, {PointSize[Medium], Red, Point @@@ perims[[All, 2]]}]

enter image description here

(* Extract the single images *)
singleimgs = ComponentMeasurements[ {comps2, img} , "MaskedImage"][[All, 2]]

enter image description here

transform[img_] := 
 Module[{img2, frame, comp, p, c, c2, cornerdemo, t, img3},
  
  (*add some padding,to make the corner detection work better*)
  img2 = ImagePad[img, 2];
  
  (*get the perimeter of the image*)
  frame = AlphaChannel[img2];
  comp = MorphologicalComponents[frame, Method -> "ConvexHull"];
  p = Flatten[
    ComponentMeasurements[comp, "PerimeterPositions"][[1, 2]], 1];
  
  (*find the corners of the image*)
  c = Sort[ImageCorners[frame, 5, 0.01]];
  
  (*the desired new corners*)
  c2 = Sort[Tuples[MinMax /@ Transpose[p]]];
  
  (*just to demonstrate the corner detection*)
  colors = {Red, Orange, Yellow, Blue};
  cornerdemo = 
   Show[frame, 
    Graphics[{Table[{colors[[n]], Disk[c2[[n]], 5]}, {n, 4}]}], 
    Graphics[Table[{colors[[n]], Circle[c[[n]], 5]}, {n, 4}]]];
  
  (*transform such that the image corners are mapped to the desired corners*)
  t = Last[FindGeometricTransform[c2, c]];
  img3 = ImagePerspectiveTransformation[img2, t, DataRange -> Full];
  
  {cornerdemo, img3}]

transform /@ singleimgs

enter image description here

$\endgroup$
7
$\begingroup$

With some of the examples given, we can bring all the images in the correct position at once, before extracting them. Here I chose 4 points using the coordinates tool.

enter image description here

imgs = CloudGet@
   CloudObject[
    "https://www.wolframcloud.com/objects/068cf32f-b753-4c38-bc01-\
f378bc64d54a"];

fourPoints = {{43.8398, 70.2148}, {336.398, 68.4375}, {342.199, 219.031}, {56.6094,209.16}}

This function processes the image. It takes the image and the four points.

processImage[img_, pts_List] := (
  rect = {{pts[[1]][[1]], pts[[1]][[2]]}, {pts[[3]][[1]], 
     pts[[1]][[2]]}, {pts[[3]][[1]], pts[[3]][[2]]}, {pts[[1]][[1]], 
     pts[[3]][[2]]}};
  transform = FindGeometricTransform[rect, pts][[2]];
  Show[fullImage = 
    Sharpen[ImagePerspectiveTransformation[img, transform, 
      PlotRange -> Full], 1], 
   Graphics[{PointSize[Large], Green, Point[rect], Transparent, 
     EdgeForm[Green], Polygon[rect]}], ImageSize -> Medium]
  )

processImage[imgs[[1]],fourPoints];

enter image description here

Now we isolate the pictures.

perim = MorphologicalPerimeter[fullImage, 60];
i2 = DeleteSmallComponents[perim];
morphBox = MorphologicalTransform[i2, {"BoundingBoxes", "Clean"}, Infinity]; 
ima=ImageMultiply[fullImage,morphBox];

enter image description here

Extract as a list of images and discard those not meeting a certain dimensions.

singles = ComponentMeasurements[ima, "MaskedImage"][[All, 2]];
validSingles = 
 Select[singles, 
  0.4 < Min[ImageDimensions[#]]/Max[ImageDimensions[#]] &]

Go through the list of images and attempt to trim the borders. It is difficult to get something that will work for all images, particularly when the border is the same color as the background of the photo. This works for some images, but not all of them.

imageList = {};
n = 1;
While[n <= Length[validSingles],
 v = validSingles[[n]];
 dim = ImageDimensions[v];
 bd = BorderDimensions[ImageAdjust[v, {-0.1, 0.1}], 0.04];
 crop1 = ImageCrop[v, 
   dim - {bd[[1]][[1]], bd[[2]][[1]]}, {Left, Bottom}];
 dim2 = ImageDimensions[crop1];
 crop2 = ImageCrop[crop1, 
   dim2 - {bd[[1]][[2]], bd[[2]][[2]]}, {Right, Top}];
 AppendTo[imageList, crop2];
 n++;
 ]
Row[imageList, "  "]

enter image description here

Better with a black background.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.