Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I need to do frame-by-frame analysis of videos. I can import the video files I need, but each file is too large to import at once. I can use the Import[fn, "Frames",{n}] command to import images one-by-one, but this takes a long time once n gets large. For example, the first frame it takes 0.14 sec, but the 200th frame takes 2.2 sec. Mathematica seems to be reading from the start of the file each time I do this. Obviously, this will not work. Is there some way to speed this up. E.g. using streams? Or tricking Mathematica to use ImageCapture to get images from a video file instead of a webcam? Thanks

share|improve this question
    
Welcome to Mathematica.SE, Vilis! I have added a little code formatting to make your question easier to read. If you would like to know more about formatting posts, see this short guide. Don't forget to mark the answer that helped you most as "accepted" by clicking the checkmark on the left-hand side of the post. –  Verbeia Jan 24 '13 at 2:12
5  
You can use ffmpeg or similar to export each frame as an image named like frame-%d.png and then you can easily import one chunk of frames at a time by reading appropriate file. –  ssch Jan 24 '13 at 2:13
    
Streams does have a certain appeal. It is to bad I don't have time to put something together for it. –  rcollyer Jan 24 '13 at 3:07
3  
What format is your video file? I tried a quick test with an AVI file and found that all frames took about the same amount of time to import. –  Todd Gayley Jan 24 '13 at 5:43
1  
You can also import a list of frames: Import[movie, {"ImageList", Range[...]}] –  cormullion Jan 24 '13 at 7:56

4 Answers 4

The performance can be improved by loading more than 100 frames at a time. To test this we can run:

LoadNFrames[n_] := (Import[video , {"Frames" , Range[1000, 1000 + n]} ]; n);

times = Table[ AbsoluteTiming[ LoadNFrames[n] ], 
         {n, {1, 2, 5, 10, 20, 30, 50, 100, 150, 200, 300, 400, 500, 600, 700, 800}}]

time it takes to load a frame VS frames loaded

One can see that in Mathematica there is a penalty for loading only few frames. We can improve the performance by preloading the video and storing it in a buffer. For instance:

buffer = Import[video, "ImageList"]; 

GetFrame[n_] := buffer[[n]]

The method uses a lot of memory, therefore, for large videos you will need to load parts of the video. I also typically cut the region of interest before putting it o the buffer.

share|improve this answer

If you only want to read it linearly you can tell ffmpeg to dump the video to stdout and then read width*height*bytes-per-pixel bytes at a time to get the video frame-by-frame:

openVideo[fname_, w_, h_] := 
 Module[{video}, 
  video["stream"] = 
   OpenRead[
    "!ffmpeg -i " ~~ fname ~~ " -loglevel quiet -f rawvideo -pix_fmt rgb24 -", 
    BinaryFormat -> True];
  video["SkipFrame", n_Integer: 1] := Skip[video["stream"], Byte, n*3*w*h];
  video["NextFrame"] := Partition[Partition[
     BinaryReadList[video["stream"], "Byte", 3*w*h]
   , 3], w];
  video["NextFrame", n_Integer] := Table[video["NextFrame"], {n}];
  video["NextImage"] := Image[video["NextFrame"], "Byte"];
  video["NextImage", n_Integer] := Table[video["NextImage"], {n}];
  video]

Here's an example:

(* Create a test movie *)
file = $TemporaryPrefix <> "testvid.avi";
Export[file,Table[Rasterize[i, ImageSize -> {352, 200}], {i, 1, 50}]];

video = openVideo[file,352,200]
video["NextImage",2]
video["SkipFrame",10]
video["NextImage"]

first

second

For some reason Mathematica seems to read through the entire video when doing Close[video["stream"]] and it might be worth to kill -15 the ffmpeg process manually to speed it up for large files.

For a 720x404 h264 mp4 video on a modest laptop it takes 13s to skip 200 frames and 0.07s to read a frame. The skipping could possibly be sped up by starting a new ffmpeg process that begins at desired frame. I would compare speeds to Import but Mathematica can't read it (for the test video Import takes .2s for first frame compared to .02s ). So as a bonus with this way you can work with many many more video formats :)

share|improve this answer

About:

This is my second answer. After playing with Mathematica's video import I have noticed that it makes import mistakes (duplicated frames + artifacts) and thus developed ffmpeg package for Mathematica which is based on ssch answer.

https://github.com/kmisiunas/ffmpeg-mathematica

The speed of this package is as follows on moderate computer:

  • Importing one frame is from 2 to 100 times slower, because current method for seeking frame is slow.
  • Importing many frames (1000+) is from 2 to 10 times faster.

I am switching to ffmpeg importer because for bulk analysis it proved to be faster and more accurate than default method.

Usage:

Mimicking Mathematica's Import function:

FFImport[ "file.avi", {"Frames", 1}] 

gives first frame (slow way).

FFImport[ "file.avi", {"Frames", Range[1,1000]}] 

gives first 1000 frames (fast)

To test status of ffmpeg library you can run FFmpeg[].

More information about ffmpeg installation in the link above.

share|improve this answer
    
@Öskå, I just updated the package for RGB support. –  Karolis May 9 at 13:09
    
Works like a charm now indeed, and it's much faster +1 –  Öskå May 9 at 13:18
    
BTW, seeking is still very slow using ffmpeg - do you have an idea how to speed it up? –  Karolis May 10 at 11:46
    
When I try to import video file using FFImport, I am getting errors BinaryReadList::strmerr: Error on stream !/usr/local/bin/ffmpeg -i "30.796.avi" -frames:v 30163 -loglevel quiet -f rawvideo -pix_fmt rgb24 -. Error message: ø½[CapitalZeta] >> General::stop: Further output of BinaryReadList::strmerr will be suppressed during this calculation. >> and it truncates mathematica. Can you help me. –  user12609 5 hours ago
    
Have you installed ffmpeg via Homebrew?see description in GitHub. –  Karolis 3 hours ago

If using Windows you can try my MathMF package (see here) which loads video frames sequentially using the Media Foundation platform via LibraryLink.

The speed is pretty good. Exact timings will vary according to CPU speed and the performance of the decoder, but as an example my PC here can read a 1280x720 WMV file at 25 frames per second and a 760x554 AVI file at 73 frames per second.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.