2
$\begingroup$

Suppouse we have a series of screenshots of a webpage, taken by hand at different scrolling positions. All they have equal width and height and random scrolling positions.

We can imitate it as follows:

session = StartWebSession["Firefox"];
WebExecute[session, "OpenPage" -> "https://en.wikipedia.org/wiki/Image_stitching"];
webpageImage0 = WebExecute[session, "CapturePage"];
height = ImageDimensions[webpageImage0][[2]];

takeNext[n_] := 
 Table[WebExecute[session, 
   "JavascriptExecute" -> 
    "window.scrollBy(0," <> ToString[RandomInteger[{-100, height-20}]] <> ")"];
  WebExecute[session, "CapturePage"], {n}]

screenshots = takeNext[30]

What would be a robust (and hopefully efficient) way to recover the complete webpage from such set of screenshots, assuming that they alltogether cover the whole page?

Currently I've developed a solution assuming that the screenshots are strongly successive. It is short but very inefficient and has a problem with removing unnecessary padding added at the botttom of produced image (I think BorderDimensions can give wrong results in some cases):

combineTwo[img1_, img2_] := 
 ImageCompose @@ 
  ImageAlign[{ImagePad[img1, {{0, 0}, {height, 0}}, White], img2}, Method -> "Fourier", 
   TransformationClass -> "Translation"]

combinedImage = Fold[combineTwo, screenshots];

finalImage = ImagePad[combinedImage, -BorderDimensions[combinedImage, 0]]

Strongly related question (but with slighlty different scope):

$\endgroup$
2
  • $\begingroup$ In version 13.0.1 there is a function: ImageStitch $\endgroup$ Commented Apr 10, 2022 at 7:47
  • $\begingroup$ @DanielHuber Yes, but apart of the fact that it is very slow, I wasn't able to get it working for such a simple case. $\endgroup$ Commented Apr 10, 2022 at 7:50

1 Answer 1

2
$\begingroup$

One possible approach using built-in methods could be the following.

  1. Using ImageCorrespondingPoints we compare the first image i1 in a list of screenshots imgs to all other images.
  2. We select i2 as the image with the maximum number of corresponding points with i1.
  3. We compute the difference between the positions (in i1 and i2) of all corresponding points and take the TrimmedMean to a given threshold th to compute a shift vector delta between i1 and and i2.
  4. We use ImageCompose and with the shift vector delta to merge i1 and i2. We remove i1 and i2 from the list of images imgs and prepend the merged image to the list of images imgs.
  5. We repeat the process until the list of images imgs has the length of one meaning that only one merged image is left.

Here two methods which implement this approach, where ImageJoinList performs all the above steps and calls ImageComposeAdd to merge two images based on a shift vector.

ImageComposeAdd[i1_,i2_,delta_]:=Module[{d1,d2,ci1,ci2,cci,cpad},
  d1=ImageDimensions[i1];
  d2=ImageDimensions[i2];

  ci1=d1*#&/@{{0,0},{1,0},{0,1},{1,1}};
  ci2=d2*#+delta&/@{{0,0},{1,0},{0,1},{1,1}};
  cci={{Min[#[[All,1]]],Min[#[[All,2]]]},{Max[#[[All,1]]],Min[#[[All,2]]]},{Min[#[[All,1]]],Max[#[[All,2]]]},{Max[#[[All,1]]],Max[#[[All,2]]]}}&@Join[ci1,ci2];
  cpad=cci-ci1;
  ImageCompose[ImagePad[i1,{{-cpad[[1,1]],cpad[[-1,1]]},{-cpad[[1,2]],cpad[[-1,2]]}},Blue],i2,delta-cpad[[1]],{0,0}]
]

ImageJoinList[imgs_List,th_:0.2,printQ_:False]/;Length[imgs]>1:=Module[{i1,i2,i2idx,iRest,IRestCPs,delta},
  i1=imgs[[1]];
  iRest=imgs[[2;;]];
  IRestCPs=ParallelMap[ImageCorrespondingPoints[i1,#]&,iRest,DistributedContexts->All];
  i2idx=Last@Last@SortBy[MapIndexed[{Length[#1[[1]]],First@#2}&,IRestCPs],First];
  i2=iRest[[i2idx]];
  iRest=Drop[iRest,{i2idx}];
  delta=TrimmedMean[First[-Differences[IRestCPs[[i2idx]]]],th];
  If[printQ,Print[MapThread[HighlightImage[#1,#2]&,{{i1,i2},IRestCPs[[i2idx]]}]~Join~{delta}]];
  ImageJoinList[{ImageComposeAdd[i1,i2,delta]}~Join~iRest,th,printQ]
]
ImageJoinList[imgs_List,th_,printQ_]/;Length[imgs]==1:=First@imgs

Here an example using a data set of 10 screenshots (generated with the code from the question) which is feed into ImageJoinList in random order:

screenshots = takeNext[10]
ImageJoinList[RandomSample[screenshots], 0.2, True]

The result of each stage of the iteration is printed. Here one example

Match

where the corresponding points are plotted over the images in pink and the shift vector is computed to be {0.,1254.}. Note that for this sage ImageCorrespondingPoints was able to find a significant number of points despite the fact, that the overlapping region is just text. The result

Result

looks pretty good upon first inspection.

The implementation is pretty rough around the edges and it is not very fast (even when running the map over ImageCorrespondingPoints in parallel). One might be able to improve the reliability and performance by playing with the Options of ImageCorrespondingPoints. I think both ImageAlign and ImageStitch use this method under hood.

ImageCorrespondingPoints is arguably a bit overkill if one really assumes, that all screenshots have the same size and it is just a matter of finding the right order and vertical shift. I tried writing something my self by overlaying the screenshots with a variable shift and minimizing the difference but I was not able to get this to run because the image overlay, trim and pad methods of Mathematica are horrible to use (or at least I am not familiar enough with them to even get these rather simple tasks running properly).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.