# Alternatives to Append/AppendTo for generating a list in an interactive DynamicModule?

I have a stack of timelapse microscopy images that I am trying to segment frame-by-frame using some of Mathematica's image processing functions. I have created a little interactive DynamicModule with a LocatorPane where the user can try out different image processing operations/parameters on a particular frame until an acceptable segmentation is arrived at for that frame. Once the segmentation for that frame is acceptable I would like the user to be able to store the generated segmentation image (aka save the "mask") and then move on to the next frame of the timelapse.

The problem I am encountering is that when I use Append or AppendTo to store each segmentation image the time to Append gets very slow as the user goes through more and more frames (I have timelapse stacks of ~100 to 200 images to go through). Is there some way to generate a list of segmented masks frame-by-frame without using Append? I have used Reap and Sow in the past when a list is generated in one fell swoop but here I am going frame-by-frame in a sort of "store-as-you-go" manner. Any suggestions as to how I might do this?

• I see another issue as well; if for whatever reason the kernel dies halfway through you will have to redo 100 images. I would save each mask to the hard drive in order to prevent this scenario and at the same time avoid the problem you are describing. – C. E. Aug 27 '14 at 17:57
• @Pickett Thanks for the suggestion. Kernel crashing has been a problem as well and I have defensively needed to adopt the "save early and often" strategy. However, I want the user to be able to review the list of segmentation masks (sorted by frame) at will as they progress to double check its accuracy so I don't want to just store to hard drive and then Clear/Remove that mask. – user13999 Aug 27 '14 at 18:02
• Perhaps list = {list, img} or mask[serno] = img; ++serno. The first will be slower to retrieve saved masks but faster to store than AppendTo. The second way might be best. There are also Association and AssociateTo, which seems better than AppendTo. – Michael E2 Aug 27 '14 at 18:19

You need to select a new data structure. As Leonid Shifrin has written

Append[list, element]


has a complexity proportional to Length[list] while

Append[association, key -> value]


is roughly constant time. Another data structure with this same property is linked lists:

{newImage, {prevImage, {prevPrevImage,{}}}


Any of these two could be used for your scenario. Fetching an image would be easier and probably faster with Association, but I will provide examples for both.

In order to build a sample list of images using linked lists, we could write

images = Nest[{Graphics[{ColorData["Rainbow"]@ RandomReal[], Rectangle[]}], #} &, {}, 100];


In order to flip through them:

fetch[list_, n_] := Part[list, Sequence @@ Append[ConstantArray[2, n - 1], 1]]
Manipulate[fetch[images, i], {i, 1, Depth[images] - 5, 1}]


(The -5 comes from a few different sources; Depth returns the required number plus one, the innermost level is empty and the Graphics object is not atomic so it is part of the expression tree. Adapt it to your type of data accordingly.)

### Using Association

Sample data, this time using Association:

images = AssociationThread[
Range[100] -> Table[
Graphics[{ColorData["Rainbow"]@ RandomReal[], Rectangle[]}],
{100}
]
];


Association is a bit easier to work with, and in order to flip through this stack you simply need to use

Manipulate[images[i], {i, 1, Length@images, 1}]


Both of these data structures should be faster than what you are currently using. Using DownValues the way kale did should have a similar performance.

Here's how I'd approach it.

Basically, use DownValues and DumpSave:

frame[1] = (*picture1*);
frame[2] = (*picture2*);

DumpSave["filepath.mx",frame]


With a fresh Kernel:

Get[filepath.mx];

Length@DownValues@frame


2

Also, Obligatory warning about DumpSave: It's platform and version specific.

• Thanks much, this approach is a big improvement over Append. – user13999 Aug 27 '14 at 20:38
• After quitting the kernel it seems my variable/definition/data (not sure what to exactly call your approach) was not saved. I think it might be because the variable I was using was a local variable (scoped as part of the DynamicModule). Any suggestions on how to save these local variable dependent definitions? I've tried Save, DumpSave and Export and can't seem to get it stored and recalled after I quit the kernel. Btw my definitions for each frame are actually a list that looks like keeperMaskList[i]={i,mask matrix for frame i,parameter used to generate mask matrix}. – user13999 Aug 29 '14 at 0:18

If you're using Mathematica 10, you can also use the new Association feature:

frame = Association[];
frame[1] = (*picture1*);
frame[2] = (*picture2*);


In some ways, associations behave more like lists than downvalues: for example, functions like Map and Select work on them directly. If you assign a different variable to an association, you'll get a copy of the association that can be modified independently (like a list).

This also seems to be slightly faster than @kale's DownValues approach:

Clear[x];
x = Association[];
(
Do[ x[i] = i^2, {i, 100000}];
Length[x]
) // Timing


0.3s

Clear[x];
(
Do[ x[i] = i^2 , {i, 100000}];
Length@DownValues[x]
) // Timing


0.4s

(But for 100-200 pictures, that probably won't make much difference.)

• The problem as I understand it is that when you add an element to a list the entire list is copied. So if the list consists of images of a total of 1GB then yes it will take some time to reallocate that memory. When you add an element to an Association you don't copy the data structure, and that is where I expect to save time. But that won't show up in this test. – C. E. Aug 27 '14 at 18:47
• @nikie I am still using version 9 but will give Association a try once I upgrade. Thanks for your suggestions! – user13999 Aug 27 '14 at 20:39