# UI performance with large image data

Some days ago, I built a small program for some of my colleagues to analyse cell images. One minor part of the user interface was the selection of the region of interest. The images are large and need to be analysed in full-size but for the selection, you don't need to see them in full-size.

The most direct way to implement this is to use the original image and only show it in a smaller size. With this the original image dimensions are preserved and I can use the original coordinates for the selection of the region. Let me give a small function that does nothing more than showing an image and a rectangular region that can be moved:

roiSelector[img_] := DynamicModule[{
pt = Round[ImageDimensions[img]/2],
dim = Round[ImageDimensions[img]/6]
},
LocatorPane[Dynamic[pt],
Show[
HighlightImage[
img,
Dynamic@{Rectangle[pt - dim, pt + dim]}
], ImageSize -> 512]
]
]

img = Import["http://biology.usf.edu/cmmb/images/cells2.jpg"];
roiSelector[img]


This selection box is far from being responsive although images that large are really common when working in science. Things like that are one reason why I believe that Mathematica is great for prototyping but doesn't scale well in real life applications. In this specific case, the user interface itself should be really fast, because it has nothing more to do than to display an image of size 512 and draw a frame above it. Although it almost looks the same, the responsiveness of a version that really uses a 512 pixel image is better

roiSelector[ImageResize[img, 512]]


Surprisingly, at least on my machine the effect can be observed when using too small images as well. The 100px version below shows the some sluggishness as well.

roiSelector[ImageResize[img, 100]]


Question: Is there better way to highlight something in large images dynamically beside the code shown above? (I have tested some other ideas myself without much success)

## Side notes:

If you change my example slightly and put the inner Dynamic@ in front of the Show and evaluate the roiSelector[img] then the whole FE becomes slow. So if you have it arranged like I do

then even editing the code in the middle becomes horribly slow. Now imagine that we loaded and displayed only one single image which is usually not the case in a real application.

My system is Mathematica 11 on OSX.

• related: 43152 – Kuba Sep 29 '16 at 15:57
• In cases like this I often wish there were a wrapper just like Dynamic but with the opposite effect, namely "whatever happens inside is only to be evaluated once and never to be looked at again". Inert would be a good name for such a feature – Sascha Sep 29 '16 at 16:19
• @Sascha The problem is, that even if you declare it as Inert, the red frame needs to be drawn over the inert image. When you move the frame, than the image needs to be redrawn on places where the red frame was. This makes it impossible to put the image as completely static because you need to access the pixels. Still the performance is really bad as internally Mathematica only needs the small version of the image to "redraw" regions. (surprisingly, on Linux, it's by far not as bad with the large image). – halirutan Sep 30 '16 at 0:41

A lot can be tweaked, but it is hardly ever straightforward:

img = Import["http://biology.usf.edu/cmmb/images/cells2.jpg"]

roiSelector[img_] := DynamicModule[
{v,
pt1, pt2,  dim,
imgDim = ImageDimensions@img,
w = 300, wI,  h = Automatic, ratio
}
,
ratio = #/#2 & @@ imgDim;
dim = Round[w imgDim/imgDim[[1]]];
pt1 = .3 dim; pt2 = .7 dim;
wI = w;

EventHandler[
Framed @ Pane[
Dynamic[
Show[
HighlightImage[
ImageResize[img, {wI, Automatic}],
{ Dynamic @ Rectangle[pt1, pt2],
Locator @ Dynamic[ pt1,
{(v = pt2 - pt1) &, (pt1 = #; pt2 = pt1 + v) &, None}
],
Locator @ Dynamic @ pt2
}
],
ImageSize -> {w, Automatic}
],
TrackedSymbols      :> {wI},
SynchronousUpdating -> False,
ImageSizeCache      -> dim
]
,
ImageSize -> Dynamic[{w, h}, ({w, h} = {1, 1/ratio} #[[1]]) &],
AppearanceElements -> All
]
,
{"MouseUp" :> ({pt1, pt2} = {pt1, pt2} w/wI; wI = w;)},
PassEventsDown -> True
]
]


Grid[{{#, #}, {#, #}}] & @ roiSelector @ img


• Your approach works well and you got my upvote. Funny enough, your answer shows exactly what's wrong with Mathematica: You can implement a feature very directly with only some lines of code, but as soon as you need it in a real application, you have to fiddle around to make it usable. – halirutan Sep 29 '16 at 23:33
• Impressive! I like the bit where you include a way to set the ImageSize with AppearanceElements-> All and then use this trick with ImageSize and Dynamic to preserve the aspect ratio. I had never seen that before, but maybe that is well known among the peeps constructing front end guis using Dynamic. But it looks like there are many more nice tricks. – Jacob Akkerboom Oct 2 '16 at 18:07
• @JacobAkkerboom Thanks :) The problem is that it does look complicated for such small task. p.s. I wasn't sure it will even work, I was surprised that MouseUp was detected even though mouse is often outside the event handler due to lag. – Kuba Oct 4 '16 at 19:42

Another approach is to set the red box to be a locator. Then you can display the image and use the red-box-locator to define the region of interest. This speeds it up a little. The major problem seems to be that the image resizing is done inside the dynamic environment. This can be helped by resizing the image once (and then using that smaller version for display). The box moves around almost instantaneously.

img = Import["http://biology.usf.edu/cmmb/images/cells2.jpg"];
box = Graphics[{Opacity[0], Rectangle[{0, 0}, {1, 1}], Red,
Opacity[0.5], Rectangle[{0.3, 0.4}, {0.6, 0.6}]}];
imgSmall = ImageResize[img, 512];
Manipulate[imgSmall, {pt, Locator, Appearance -> box}]

• +1 for the idea but the approach itself is as slow as using HightlightImage. Basically, I don't want to scale the image for display because there are a lot of examples, where this isn't possible. – halirutan Sep 29 '16 at 23:50
• @halirutan It's always possible to scale the image for display. Indeed, it is needed since the display is only of finite resolution. The only question is where it is done and how often. Mathematica just might not be a performant graphics framework. Its strength are probably somewhere else. – Trilarion Sep 30 '16 at 11:00