# Using mask to segment growing cells over multiple timepoints

I am trying to segment some very irregular cells out of images generated by timelapse microscopy. These cells don't fit any simple shape (not circular or elliptical) and they grow and change shape over time. I am trying to segment them by creating a manually generated mask for the first timepoint and then using that mask to find the cell in later timepoints and updating the mask to fit the new shape of the cell (since the cell will have changed shape due to growth and possibly changed position slightly over time). In other words, the mask will need to be dynamically modified over time.

Here's an image of a field of the crazy looking cells I am trying to segment:

Here's a manually generated mask for cell in the lower right I would like to get data on:

Here are the same cells slightly later (cells are a little bigger and have shifted a bit). I would like to use my mask above to find the same cell in this later image and then modify the original mask to fit the cell here (so I can get data on its growth without having to manually draw a new mask):

I realize this is not a simple problem but was hoping to at least be able to locate the same cell at the later timepoint using ImageCorrelate or some shape detecting function. However, I am not having much luck with this. Any suggestions are welcome.

Not perfect, but a start.

(*Load the Images*)

i1 = ImageResize[Import["http://i.stack.imgur.com/rcKOy.jpg"], 600];
im = ImageResize[Import["http://i.stack.imgur.com/2keNS.jpg"], 600];
i2 = ImageResize[Import["http://i.stack.imgur.com/IuYU8.jpg"], 600];

(* get the corresponding points*)

ivp = ImageCorrespondingPoints[i1, i2];
ivpt = Round@Transpose@ivp;

(* Select the corresponding points inside the mask *)

ivps = Select[ivpt, MemberQ[PixelValuePositions[Binarize@im, 1], #[[1]]] &];

Show[i1, Graphics[{Red, Point[Transpose[ivps][[1]]]}]]
Show[i2, Graphics[{Red, Point[Transpose[ivps][[2]]]}]]


Now we will try to find a pure quadratic (for simplicity) transformation between the points.

dataXY = First@Transpose@ivps;
{dataFx1, dataFy1} = Map[List, Transpose@Last@Transpose@ivps, {2}];
ones = ConstantArray[{1}, Length@dataFx1];
d1 = Flatten /@ Transpose[{dataXY, ones, dataFy1}];
d0 = Flatten /@ Transpose[{dataXY, 0 ones, dataFx1}];
data = Join[d0, d1];

modelx[x_, y_] := ax x^2 + bx x y + cx y^2 + ix
modely[x_, y_] := ay x^2 + by x y + cy y^2 + iy
model[x_, y_, s_] := modely[x, y]*s + modelx[x, y]*(1 - s)

ff = FindFit[data, model[x, y, s], {ax, ay, bx, by, cx, cy, ix, iy}, {x, y, s}];


it = ImageForwardTransformation[EdgeDetect[im, 2],
({modelx @@ ##, modely @@ ##} /. ff) &,
DataRange -> Full]


And superimposed with the second image we get:

ImageMultiply[ i2, ColorReplace[ColorNegate@it, Black -> Red]]


• Thanks for giving this a try. Took me a bit to get a sense of what's going on in the code. Couple questions: i.) Is it possible to get a dense or possibly uniform sampling of points within the mask to then use ImageCorrespondingPoints on in the later image? It's unclear to me how the ImageCorrespondingPoints algorithm finds the corresponding points it does. ii.) Do you think fitting a global set of corresponding points from the whole of both images would give a better fit? iii.) What other models (besides quadratic) do you think are worth a shot trying out here? – user13999 Mar 11 '16 at 20:38

It looks as if the cells are dark and have a bright border. If that is always the case, and if you don't mind a little manual adjustment, you can put a marker on each cell and use watershed segmentation to find the bright borders in between.

I've manually put points in each of the cells using a LocatorPane:

img = Import["http://i.stack.imgur.com/rcKOy.jpg"];
pts = {{399, 496}, {497, 537}, {346, 507}, {349, 434}, {434,
269}, {337, 385}, {213, 485}, {231, 443}, {196, 536}, {79,
571}, {98, 505}, {150, 434}, {128, 397}, {576, 558}, {536,
581}, {198, 581}, {24, 540}, {62, 525}, {92, 87}};

LocatorPane[Dynamic[pts], img,
Appearance -> {Graphics[{Red, Point[{0, 0}]}]},
LocatorAutoCreate -> True]


And these are the resulting watershed boundaries:

HighlightImage[img,
ColorNegate@Binarize@Image[WatershedComponents[img, pts]]]


Since you want to track these cells in a series of frames, you could then find a new marker point for each cell, and use that marker point in the next frame. The marker should be as far away as possible from the border, so I would use ComponentMeasurements to get a Mask of each component, then use DistanceTransform on the mask and find the point with the highest value, i.e. the point that's farthest from the boundary.

• This looks pretty good. I have actually tried watershedding before but you seem to be getting better results than I did. Couple follow-ups: – user13999 Mar 11 '16 at 15:08
• Couple follow-ups: i.) how can I split a watershed contour when the watershed results in undersegmentation? Ideally I would like to split the contour manually by drawing a line across a contour or at worst split the contour by adding another marker and rerunning the watershed. ii.) how did you find the markers you used for this position? Trial and error or did you do the distance transform? – user13999 Mar 11 '16 at 15:16
• @user13999: The only way to split watersheds is to use two markers instead of one. It shouldn't matter too much where in the two cells you place the markers. I placed the markers manually using LocatorPane. I can't tell if it worked immediately because it's so robust or because I got lucky. But as long as the border around/between the cells is brighter than anything in the cells, it should be robust. – Niki Estner Mar 11 '16 at 15:29
• PS: I tried the distance transform to find improved markers for the next frame, but wasn't overwhelmed by the results. I'll see if I can think of something else when I have a little more time... – Niki Estner Mar 11 '16 at 15:30
• Yes, when I've tried to work with watershedding in the past the problem is that it needs a lot of manual intervention to get the markers just right (especially for a crowded field of cells). If you have a lot of time and patience you can usually get a pretty good segmentation result but ideally the choice of marker and position could be automated and informed by the previous timepoint segmentation. – user13999 Mar 11 '16 at 19:08