Video segmentation

Question

Mathematica v12.1 introduced Video and various functions like VideoFrameMap and VideoExtractFrames. I want to mask out the background and extract the flag in this video: https://www.pexels.com/video/great-britain-flag-hanging-on-a-pole-3150358

So far I have loaded the video and created a rough rectangular mask.

vid = Video["video.mp4"];
firstFrame = VideoFrameList[vid, 1][[1]];
mask = ConstantArray[0, Reverse@ImageDimensions@firstFrame];
mask[[900 ;; 1250, 1500 ;; 2200]] = 1;
mask = Image@mask;
HighlightImage[firstFrame, mask]

I tried to grow the mask to fill the whole flag with RegionBinarize but this is not effective. I want to extract the flag in all frames and produce a video of just the flag on a black background. I thought it might be possible to turn the video into a volume with Image3D and try to do the segmentation there. Any ideas?

Update: I have a hand drawn mask for the first frame if it helps:

Answer 1

I had to scale the video (and the mask) down to 480p as Mathematica couldn't handle the image processing at high resolution. My approach below naively does a segmentation on each frame and this causes poor temporal coherence in the result leaving holes and flickering artifacts. It's also very slow. I am looking at alternative solutions that overcome these problems, but for now here's what I've achieved:

The code below works as follows on each frame independently:

Training:

1. Convert the training frame to the LAB colour space.

2. Perform a Watershed transform to finely dice up the frame into small regions.

3. For each region, get the proportion of the mask in the region. If it exceeds a threshold, then it's in class 1, otherwise class 0

4. For each region, get some features about the region, e.g mean colour, standard deviation, centroid position, number of pixels.

5. Form training data from steps 3 and 4 in the form of a list of features -> class and use Classify to create a ClassifierFunction

Segmentation of non-training frames:

1. Convert the frame to LAB.

2. Get a Watershed transform of the frame as in Training step 2.

3. As in step 4 of the Training phase, get features for each region.

4. Use the ClassifierFunction we created earlier to classify regions in the frame as 1 (in the mask) or 0 (outside of the mask)

5. Replace the regions in the Watershed transform image with the mask values from step 4, and ImageMultiply with the frame to apply the mask.

getSegmentation[img_] := Image[WatershedComponents[img, Method -> {"MinimumSaliency", 0.3}]]

pixelFeatures[pixels_] := 
 Join[Mean[pixels], If[Length@pixels > 1, StandardDeviation[pixels], {0, 0, 0}]]
 (* could also use a histogram here however it performs quite badly *)

getFeatures[img_, segmentation_] := ParallelTable[
  With[{pos = PixelValuePositions[segmentation, i]},
   Join[Mean[pos], {Length@pos}, 
    pixelFeatures@PixelValue[img, pos]]], {i, 1, 
   Round@Max@segmentation}]

getSegmentLabelling[mask_, segmentation_, threshold_] :=
 (* if more than threshold many pixels of each component appear in the mask, label this piece as flag, else background *)
 ParallelTable[
  If[Mean[PixelValue[Binarize@mask, 
      PixelValuePositions[segmentation, i]]] > threshold, 1, 0], {i, 1, Round@Max@segmentation}]

createTrainingData[frame_, mask_] := 
 Module[{seg = getSegmentation[frame]},
  Return[Thread[
    Rule[getFeatures[frame, seg], getSegmentLabelling[mask, seg, 0.5]]]]
  ]

applyClassifier[cf_, frame_] := Module[{seg = getSegmentation[frame]},
  (* I would use ColorReplace here but it doesn't work *)
  Image[Round[ImageData[seg]] /. MapIndexed[First[#2] -> cf[#1] &, getFeatures[frame, seg]]]]

(* use it *)
vid = Video["flag.mp4"];
firstFrame = VideoFrameList[vid, 1][[1]];
mask = Binarize@ImageResize[Import["firstFrameMask.jpg"], ImageDimensions[firstFrame]];
HighlightImage[firstFrame, mask]
trainingData = createTrainingData[ColorConvert[firstFrame, "LAB"], mask];
classifier = Classify[trainingData];

result = ImageMultiply[#, applyClassifier[classifier, ColorConvert[#, "LAB"]]] & /@ 
  Table[VideoExtractFrames[vid, i], {i, 1, 17, 0.25}]

Answer 2

With the scaled down 480p video, cutting out a rough box ROI around the flag and doing a BrightnessEqualize on a volume of frames (i.e Image3D) followed by RemoveBackground works quite nicely for me and it's much simpler than my other segmentation approach:

framevolume = Image3D[VideoFrameList[vid, 90]];
ListAnimate[
 Image3DSlices@
  RemoveBackground[
   BrightnessEqualize@
    ImageTake[framevolume, {1, -1}, {70, 170}, {160, 280}], {"Background", {Cyan, .31}}]]