Detect and correct sheared squares in image

Question

I have the image

I want to detect the plots and warp them into perfect squares. My workflow so far:

corners=ImageCorners[img,10];

This returns a lot of corners, and I find I have to go through them manually. I would really appreciate a heuristic to find the proper corners automatically, maybe some filter based on the amount of whitespace in a big (50 px) smoothed area around a pixel. To manually select corners:

HighlightImage[img,MapIndexed[Labeled[#1,ToString@First@#2]&,corners[[;;200]]]]
(*no good way to layout that many labels,
is there a way to dynamically make the labels near the mouse pointer big?*)
HighlightImage[img,MapIndexed[Labeled[#1,ToString@First@#2]&,
corners[[Join[{15,171,89,108},{77,113,79,158}]]]]]

I don't know if those indices will be the same on every machine. Finally, the warping

sketch0=ImageTake[ImageTransformation[img,Last@FindGeometricTransform[
 corners[[{15,171,89,108}]],{20,20}+#&/@{{0,#},{0,0},{#,0},{#,#}}&@300],
 ImageDimensions@img,DataRange->Full],All,{0,340}]
sketch1=ImageTake[ImageTransformation[img,Last@FindGeometricTransform[
 corners[[{77,113,79,158}]],{20,20}+#&/@{{0,#},{0,0},{#,0},{#,#}}&@300],
 ImageDimensions@img,DataRange->Full],All,{0,340}]

I'm not super concerned about the dewarping code. It's ugly, but does the trick.

Feedback on my code is appreciated, as well as less tedious methods for detecting features. In my mind a perfect program to do this would highlight the closest strong corner to the mouse as it moves across the image so you could quickly select the relevant features -- a module like this is feasible in Mathematica.

Answer 1

Feedback on my code is appreciated, as well as less tedious methods for detecting features. In my mind a perfect program to do this would highlight the closest strong corner to the mouse as it moves across the image so you could quickly select the relevant features -- a module like this is feasible in Mathematica.

If I understood, maybe this will help for a manual but quicker method of identifying the corners.

You can use the image edit Coordinate tool , and copy as a list

I actually think your attempt is really nice, I'm not sure you'll get it any more 'square'.

Maybe trying to recreate the data within the square and plotting could be interesting

With the same method:

You can start to rebuild by taking the coordinates and indices

or follow the methods to recreate in this post

Answer 2

To extract well separated sub images from a composite image, we can proceed as follows:

I assume that the sub images are arrange horizontally with a clear gap, like e.g.:

We first make the image into black and white without gray. Then we transform it into an array of 0 and 1 (0 corresponds to black, 1 to white):

im = ImageAdjust[im0];
im = Binarize[im, .9];
xy = MorphologicalComponents[im];

Next, we determine the positions of the zeros. Then we determine the horizontal positions of the pixels without duplicates (note the graphics coordinates x/y are the array coordinates y/x ).

pos = Position[xy, 0];
posx = Union@pos[[All, 2]];

As the partial images are well separated along the horizontal axes, we can split the x positions into continuous pieces:

posx = MinMax /@ Split[posx, #2 - #1 < 5 &];

Next, we determine the min/max y coordinates of the different x- pieces:

sely[int_] := 
  MinMax[Select[pos, IntervalMemberQ[Interval[int], #[[2]]] &][[All, 1]]];
posy = sely /@ posx;

We now have the extremal x/y coordinates of the sub images and can extract these sub images:

images=(OperatorApplied[ImageTake, 3][im0]) @@@ Transpose[{posy, posx}]