3
$\begingroup$

Mathematica v12.1 introduced Video and various functions like VideoFrameMap and VideoExtractFrames. I want to mask out the background and extract the flag in this video: https://www.pexels.com/video/great-britain-flag-hanging-on-a-pole-3150358

So far I have loaded the video and created a rough rectangular mask.

vid = Video["video.mp4"];
firstFrame = VideoFrameList[vid, 1][[1]];
mask = ConstantArray[0, Reverse@ImageDimensions@firstFrame];
mask[[900 ;; 1250, 1500 ;; 2200]] = 1;
mask = Image@mask;
HighlightImage[firstFrame, mask]

flag

I tried to grow the mask to fill the whole flag with RegionBinarize but this is not effective. I want to extract the flag in all frames and produce a video of just the flag on a black background. I thought it might be possible to turn the video into a volume with Image3D and try to do the segmentation there. Any ideas?

Update: I have a hand drawn mask for the first frame if it helps: first frame mask

$\endgroup$
2
$\begingroup$

I had to scale the video (and the mask) down to 480p as Mathematica couldn't handle the image processing at high resolution. My approach below naively does a segmentation on each frame and this causes poor temporal coherence in the result leaving holes and flickering artifacts. It's also very slow. I am looking at alternative solutions that overcome these problems, but for now here's what I've achieved:

flag anim

The code below works as follows on each frame independently:

Training:

  1. Convert the training frame to the LAB colour space.

  2. Perform a Watershed transform to finely dice up the frame into small regions.

  3. For each region, get the proportion of the mask in the region. If it exceeds a threshold, then it's in class 1, otherwise class 0

  4. For each region, get some features about the region, e.g mean colour, standard deviation, centroid position, number of pixels.

  5. Form training data from steps 3 and 4 in the form of a list of features -> class and use Classify to create a ClassifierFunction

Segmentation of non-training frames:

  1. Convert the frame to LAB.

  2. Get a Watershed transform of the frame as in Training step 2.

  3. As in step 4 of the Training phase, get features for each region.

  4. Use the ClassifierFunction we created earlier to classify regions in the frame as 1 (in the mask) or 0 (outside of the mask)

  5. Replace the regions in the Watershed transform image with the mask values from step 4, and ImageMultiply with the frame to apply the mask.

getSegmentation[img_] := Image[WatershedComponents[img, Method -> {"MinimumSaliency", 0.3}]]

pixelFeatures[pixels_] := 
 Join[Mean[pixels], If[Length@pixels > 1, StandardDeviation[pixels], {0, 0, 0}]]
 (* could also use a histogram here however it performs quite badly *)

getFeatures[img_, segmentation_] := ParallelTable[
  With[{pos = PixelValuePositions[segmentation, i]},
   Join[Mean[pos], {Length@pos}, 
    pixelFeatures@PixelValue[img, pos]]], {i, 1, 
   Round@Max@segmentation}]

getSegmentLabelling[mask_, segmentation_, threshold_] :=
 (* if more than threshold many pixels of each component appear in the mask, label this piece as flag, else background *)
 ParallelTable[
  If[Mean[PixelValue[Binarize@mask, 
      PixelValuePositions[segmentation, i]]] > threshold, 1, 0], {i, 1, Round@Max@segmentation}]

createTrainingData[frame_, mask_] := 
 Module[{seg = getSegmentation[frame]},
  Return[Thread[
    Rule[getFeatures[frame, seg], getSegmentLabelling[mask, seg, 0.5]]]]
  ]

applyClassifier[cf_, frame_] := Module[{seg = getSegmentation[frame]},
  (* I would use ColorReplace here but it doesn't work *)
  Image[Round[ImageData[seg]] /. MapIndexed[First[#2] -> cf[#1] &, getFeatures[frame, seg]]]]

(* use it *)
vid = Video["flag.mp4"];
firstFrame = VideoFrameList[vid, 1][[1]];
mask = Binarize@ImageResize[Import["firstFrameMask.jpg"], ImageDimensions[firstFrame]];
HighlightImage[firstFrame, mask]
trainingData = createTrainingData[ColorConvert[firstFrame, "LAB"], mask];
classifier = Classify[trainingData];

result = ImageMultiply[#, applyClassifier[classifier, ColorConvert[#, "LAB"]]] & /@ 
  Table[VideoExtractFrames[vid, i], {i, 1, 17, 0.25}]
| improve this answer | |
$\endgroup$
1
$\begingroup$

With the scaled down 480p video, cutting out a rough box ROI around the flag and doing a BrightnessEqualize on a volume of frames (i.e Image3D) followed by RemoveBackground works quite nicely for me and it's much simpler than my other segmentation approach:

framevolume = Image3D[VideoFrameList[vid, 90]];
ListAnimate[
 Image3DSlices@
  RemoveBackground[
   BrightnessEqualize@
    ImageTake[framevolume, {1, -1}, {70, 170}, {160, 280}], {"Background", {Cyan, .31}}]]

flag animation from removing background

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.