# Extract timestamp of specific frames in video

I would like to do the following : I have a set of slides and a video in which those slides are discussed. Now I would like to extract the timestamp when the slide appears in the video. I would suggest to proceed in the following way:

1. Import the video Import[] (How can I import a mp4.file ?)

2. Import the slides as images

3. For all frames in the video, I define the following function: I compare a certain amount of pixels of the frame to all the slides images. When a certain percentage of pixels are the same, I return the slide image and the corresponding timestamp.

My questions:

1. How can I extract all timestamps for a video frames ?
2. How can I extract the values of pixels within a certain geometric region of the frame.

In my video, a person might stand in fron of the slides, therefor I cannot simply say that I detect a slide in the video, if all pixels of the frame and the slide are the same.

Thanks.

Here is some material to try:

Instead of a video, one can use a gif:

One can try to extract the timestamp of this frame:

And if someone is steading in front (please excuse my drawing skills):

Please try with the following material: YOu can find the frames and the slides to match. Aim: Find position of the slides within the frame list.

https://www.dropbox.com/sh/l1deic1ris2il6w/AAAOU_ICZM_f0T-9M9kfNNeRa?dl=0

• You are trying to build a en.wikipedia.org/wiki/Content-based_image_retrieval system . That article should be a starting point for finding much more reliable techniques for comparing images than what you describe. (Comparing the pixel colors directly is extremely brittle for any comparison of real-world-to-synthetic (or other real-world-to-real-world) images. It only sort-of works for two synthetic images.) – masterxilo Oct 23 '17 at 19:24
• LibraryLink/tutorial/ImageProcessing#263719837 ( reference.wolfram.com/language/LibraryLink/tutorial/… ) "Importing Frames from a Video" from the documentation might help too. – masterxilo Oct 23 '17 at 19:27
• @masterxilo Thank you very much for the two very useful links ! – james Oct 24 '17 at 6:13

First I have to say that I'm a bit skeptical whether what you want to do can work in general. What if the person blocks everything that is unique of that slide? What if some slides look the same?

But let's ignore those possible problems for now and try a very simple approach. This is only meant as a starting point! My answer is based on this very good answer here and this one, which contains more explanation and ways to improve.

First let us import your example data:

gif = Import["https://i.stack.imgur.com/k4ChI.gif"];


We scale the images down to 32x32 and obtain the pixel data using ImageData. Scaling down will increase the robustness against small differences between the slide we are searching for and the video, as well as decrease the computation time. Note that you could probably scale down the whole video beforehand. We search for the frame with the head on it, change this line to search for the other one if you want to try it.

seeked = Flatten[ImageData@ImageResize[framewithhead, {32, 32}], 1];
small = Flatten[ImageData[ImageResize[#, {32, 32}]], 1] & /@ gif;


In order to decide whether two colors are similar we can define the following function. Play around with the threshold value!

SimilarColor[a_, b_] := If[Total[(a - b)^2] < 0.0005, 1, 0];


Now we just pick the frame with the highest score, i.e. the highest number of sectors that are similar to the frame we are looking for.

score = Total@MapThread[SimilarColor, {#, seeked}] & /@ small;
Position[score, Max[score]]


This returns 15 in both cases (with and without head)!

Edit: for the provided slides

Lets rename your slides, such that {{1}} became {{001}}. You can do it with something like

Do[RenameFile[
NotebookDirectory[] <> "\\frames\\{{"  <> ToString[i] <> "}}.jpg",
NotebookDirectory[] <> "\\frames\\{{0" <> ToString[i] <> "}}.jpg"], {i, 10, 99}]


Now we import all those images, for example:

frames = Import[#] & /@
FileNames["*.jpg", NotebookDirectory[] <> "\\frames"];
slides = Import[#] & /@
FileNames["*.jpg", NotebookDirectory[] <> "\\slides"];


We can scale down the images, I used a slightly higher resolution because of some details in your slides.

res = 48;
smallframes =
Flatten[ImageData[ImageResize[#, {res, res}]], 1] & /@ frames;
smallslides =
Flatten[ImageData[ImageResize[#, {res, res}]], 1] & /@ slides;


I also slightly changed the color comparison, I'm not actually sure that you need to change it, but why not try a different one :)

SimilarColor[a_, b_] := If[And @@ ((# < 0.05) & /@ ((a - b)^2)), 1, 0];


Now comes the heavy calculation (takes a couple of minutes on my laptop): We label each frame by the slide that's in the background.

labels = Monitor[Table[
With[{score =
Total@MapThread[SimilarColor, {#, smallframes[[i]]}] & /@ smallslides},
Position[score, Max[score]]][[1, 1]], {i, Length[frames]}], i]


{2,2,2,2,2,2,2,2,2,2,2,2,2,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,6,6,6,6,6,6,6,6,6,6,6,7,7,7,7,7,7,7,7,7,8,8,8,8,8,8,8,8,9,9,9,9,9,9,10,10,10,10,10,10,10,10,10,10,10,11,11,11,11,12,12,12,13,13,13,13,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,14,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,20,20,20,20,20,20,20,20,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,3,3,3,3,3,3,3,3,3,3}

This list should contain all the information you need, in particular the first occurrence of slide 9 (with the photo mask) is

 Min@Position[labels, 9]


95

• Thank you very much for this answer ! One question: Why do you scale down the image to 32 x 32 ? – james Oct 20 '17 at 6:36
• The number 32 is pretty arbitrary. There's a trade-off between accuracy of the method (large resolution) and tolerance with respect to small changes (e.g. moved/rotated a little bit) (small resolution). I assume that the slides in your video will not match the slides you are looking for pixel by pixel. The value of 32 gives reasonable robustness and picks the correct frame. Play with the value! Also the computing time depends on the scaling factor. – M. Stern Oct 20 '17 at 14:28
• Can you try with the files provided ? – james Oct 25 '17 at 23:18
• yes it seems to work. I chose resolution 48x48 (because there are quite small differences in your slides) and adopted the color comparison to SimilarColor[a_, b_] :=If[And @@ ((# < 0.05) & /@ ((a - b)^2)), 1, 0];. I will edit my answer accordingly – M. Stern Oct 26 '17 at 1:35
• You are awesome !! Thank you so much !! Maybe you can also help me with formating the result ? mathematica.stackexchange.com/questions/158664/… – james Oct 26 '17 at 14:42

Import the video Import[] (How can I import a mp4 file?)

Often, MP4 files can be imported as AVI. Another way is to wrap the MP4 file with QuickTime without transcoding, and import the MOV file.

Get attributes of the video file with:

{bitDepth, colorSpace, duration, frameCount, frameRate, imageSize,
videoEncoding} =
Import["file.mp4", {"AVI", {"BitDepth", "ColorSpace", "Duration",
"FrameCount", "FrameRate", "ImageSize", "VideoEncoding"}}]

(*{8,"RGB",421.467,12646,30.0047,{1280,720},"H.264"}*)


How can I extract all timestamps for the video frames?

To find the timestamp for a frame that matches a slide, subtract 1 from the frame number and divide by the frame rate, which gives the time in seconds:

timeStamp[frame_] := (frame - 1)/frameRate
timeStamp[1000]

(*33.2947*)


## How to search frames more quickly

For a large video file, importing every frame is a slow process. One way to avoid importing the entire video is to use a sample of frames. For example, here's how to sample 12 frames from a video.

Import["file.mp4", {"AVI","ImageList", Range[1, frameCount, Round[frameCount/12]]}]


Using Range to choose only some of the video frames, the video can be easily sampled and tested using M. Stern's method. This range:

Range[1, frameCount, Round[frameRate/2]]


samples frames approximately every 1/2 second, and

Range[1, frameCount, Round[2*frameRate]]


samples at approximately 2-second intervals.

Assuming the individual slides in the video change about 20 seconds per slide, a 2- or 3-second interval should allow enough frames to test. Adjust the sample interval depending on the length of the video and how often the slides change.

Avoid testing parts of the video that don't have slides. If there's a 20-second lead-in before the first slide, start Range at Round[20*frameRate] instead of frame 1.

There's a trade-off between sample interval and how accurately a slide change will be timestamped. A large interval tests fewer frames, but the timestamp might miss the exact time that a slide change happens.

• Thanks a lot! Do you know how I can read each frame of a video file to compare the pixels to a given image in a very fast way ? The import of frames takes ages ! :( – james Oct 22 '17 at 8:59
• @totyped Sampling frames will work if the video is not too long and each frame's dimensions aren't too large. Otherwise, it may be necessary to partition the video into blocks of frames, and test block-by-block. – creidhne Oct 22 '17 at 22:00
• Thank you very much for your edit ! – james Oct 24 '17 at 5:20
• Please try with the material provided to the question... I was not able to do it. – james Oct 25 '17 at 23:17