Alternatives to Append/AppendTo for generating a list in an interactive DynamicModule?

Question

I have a stack of timelapse microscopy images that I am trying to segment frame-by-frame using some of Mathematica's image processing functions. I have created a little interactive DynamicModule with a LocatorPane where the user can try out different image processing operations/parameters on a particular frame until an acceptable segmentation is arrived at for that frame. Once the segmentation for that frame is acceptable I would like the user to be able to store the generated segmentation image (aka save the "mask") and then move on to the next frame of the timelapse.

The problem I am encountering is that when I use Append or AppendTo to store each segmentation image the time to Append gets very slow as the user goes through more and more frames (I have timelapse stacks of ~100 to 200 images to go through). Is there some way to generate a list of segmented masks frame-by-frame without using Append? I have used Reap and Sow in the past when a list is generated in one fell swoop but here I am going frame-by-frame in a sort of "store-as-you-go" manner. Any suggestions as to how I might do this?

Answer 1

You need to select a new data structure. As Leonid Shifrin has written

Append[list, element]

has a complexity proportional to Length[list] while

Append[association, key -> value]

is roughly constant time. Another data structure with this same property is linked lists:

{newImage, {prevImage, {prevPrevImage,{}}}

Any of these two could be used for your scenario. Fetching an image would be easier and probably faster with Association, but I will provide examples for both.

Using linked lists

In order to build a sample list of images using linked lists, we could write

images = Nest[{Graphics[{ColorData["Rainbow"]@ RandomReal[], Rectangle[]}], #} &, {}, 100];

In order to flip through them:

fetch[list_, n_] := Part[list, Sequence @@ Append[ConstantArray[2, n - 1], 1]]
Manipulate[fetch[images, i], {i, 1, Depth[images] - 5, 1}]

(The -5 comes from a few different sources; Depth returns the required number plus one, the innermost level is empty and the Graphics object is not atomic so it is part of the expression tree. Adapt it to your type of data accordingly.)

Using Association

Sample data, this time using Association:

images = AssociationThread[
   Range[100] -> Table[
     Graphics[{ColorData["Rainbow"]@ RandomReal[], Rectangle[]}],
     {100}
     ]
   ];

Association is a bit easier to work with, and in order to flip through this stack you simply need to use

Manipulate[images[i], {i, 1, Length@images, 1}]

Both of these data structures should be faster than what you are currently using. Using DownValues the way kale did should have a similar performance.

Answer 2

Here's how I'd approach it.

Basically, use DownValues and DumpSave:

frame[1] = (*picture1*);
frame[2] = (*picture2*);


DumpSave["filepath.mx",frame]

With a fresh Kernel:

Get[filepath.mx];

Length@DownValues@frame

2

Also, Obligatory warning about DumpSave: It's platform and version specific.

Answer 3

If you're using Mathematica 10, you can also use the new Association feature:

frame = Association[];
frame[1] = (*picture1*);
frame[2] = (*picture2*);

In some ways, associations behave more like lists than downvalues: for example, functions like Map and Select work on them directly. If you assign a different variable to an association, you'll get a copy of the association that can be modified independently (like a list).

This also seems to be slightly faster than @kale's DownValues approach:

Clear[x];
x = Association[];
(
  Do[ x[i] = i^2, {i, 100000}];
  Length[x]
  ) // Timing

0.3s

Clear[x];
(
  Do[ x[i] = i^2 , {i, 100000}];
  Length@DownValues[x]
  ) // Timing

0.4s

(But for 100-200 pictures, that probably won't make much difference.)