How to track a deformable rod in video?

I have a video of nanotubes in solution taken with a confocal laser microscope. I typically track these nanotubes with Fiesta or Fiji, but in this case, there is variation in brightness that is confounding my tracking programs.

Here’s a link to the full video (.tif file)

What I tried

frames = CloudGet @ CloudObject @ "https://www.wolframcloud.com/obj/b4056a15-97e5-464a-879e-89852b85ebfd";


Using the image tool-ribbon to get points in the target blob:

So then extract the region of the nanotube as a mask:

pts = {{104.125, 63.28125}, {89.1328125, 66.29296875}};
RegionBinarize[frames[[1]], pts, .6]


But then using ImageCorrespondingPoints and ImageFeatureTrack the results were pretty bad:

pos = ImageValuePositions[RegionBinarize[frames[[1]], c, .6], 1];
res = ImageFeatureTrack[frames, pos]
Graphics[{If[FreeQ[#, _Missing], {RandomColor[], Line[{#}]}] & /@ Transpose[res]}]


g = If[FreeQ[#, _Missing], Graphics@Point@#, Nothing] & /@ res;
ListAnimate@g


Treating everything as points doesn't really help in recovering the perimeter:

I was hoping some cv experts could help out with tracking the deforming shape of the nanotube over all the frames.

Since the object you want to track is moving slowly, you can use ImageDisplacements to calculate a per-pixel optical flow:

f = frames[[200 ;; ;; 3]];
flow = Image /@ ImageDisplacements[f];


Now flow contains a dx/dy displacement vector for each pixel for each pair of frames, that tells you the relative displacement between two frames. $$*$$

So the idea would be to pick a starting point in the first frame:

pt = {83., 48.}
HighlightImage[f[[1]], pt]


and then track that point through from frame to frame by accumulating the relative displacement vectors:

pts = FoldList[
Function[{p, flow}, p + Mean[ImageTrim[flow, p, 10]]],
pt, flow];

Manipulate[HighlightImage[f[[i]], {pts[[i]]}], {i, 1, Length[f], 1}]


$$*$$ Note 1: I've skipped the first 200 frames, because there's an abrupt jump around frame 100. It's much larger than 1 pixel, so the dense flow "looses" the object here. You would have to use e.g. image alignment whole frames to remove this "jump" first.

Note 2: I'm only using every third frame to make processing faster.

• I'm trying to find the full segmentation boundary of the string shape though (not just a point) – M.R. Oct 18 '19 at 20:16
• Because what I really need to compute how it "wiggles" you see... – M.R. Oct 18 '19 at 20:19
• @M.R. Can you define "how it wiggles" a bit more? I suspect it would be more robust to estimate "wiggliness" from the optical flow directly, instead of tracking points and then calculating something from these points. Tracking usually means that errors accumulate from one frame to the next. When you calculate a mean over some function of the flow vectors, errors (might hopefully) average out. – Niki Estner Oct 19 '19 at 5:39
• I need to model it with the difeq for rod deformation and look at the normal modes. – M.R. Oct 20 '19 at 21:27
• @M.R. Why not, then, track multiple points? Maybe one could do something like the perimeter of a region that is the same shape as the blob? – CA Trevillian Oct 21 '19 at 1:54