Importing videos in Mathematica

Question

Mathematica's Import command is purportedly able to import .AVI files. However, I find that many files that I want to import do only result in MMA showing a blank frame or another placeholder. All or some of my video-players are able to show these AVI files.

This is an example where it works as advertised:

Import["http://er.jsc.nasa.gov/seh/jfkrice.avi", {"ImageList", 2}]

However,

Import["http://people.sc.fsu.edu/~jburkardt/data/avi/ccvt_box.avi", {"ImageList", 2}]

doesn't work (VLC does). I get an empty white frame.

More examples:

Import["http://people.sc.fsu.edu/~jburkardt/data/avi/sine_wave.avi", {"ImageList", 2}]

(works on VLC and mediaplayer classic)

Import["http://people.sc.fsu.edu/~jburkardt/data/avi/star_collapse.avi", {"ImageList", 2}]

BIG-126 MB!!
Should have looked like this: and works on Windows Media Player, VLC and mediaplayer classic, but I get a white box in MMA.

This is on windows 7-64/MMA8.04. I have the k-lite codecs pack installed.

---

UPDATE

Responding to Thomas' comment below I found most of the sample files I linked to in my original post gone. I tried to gather a new set and found some that worked and some that don't:

Importable:

Import[#, "VideoEncoding"] & /@ 
 {"http://er.jsc.nasa.gov/seh/jfkrice.avi", 
  "http://redmine.yorba.org/attachments/615/MVI_0572.AVI",
  "http://www-eng-x.llnl.gov/documents/a_video.avi",
  "http://redmine.yorba.org/attachments/628/MVI_4981.AVI",
  "http://www.csoft.co.uk/video/original/earth.avi",
  "http://www.mysticfractal.com/video/fp.avi", 
  "http://www.softage.ru/files/video-codec/uncompressed/suzie.avi", 
  "http://archive.org/download/Architects_of_Tomorrow/2007-12-10-02-39-00.avi"}

{"msvc", "MJPG", "msvc", "MJPG", "msvc", "msvc", "YUV", "Uncompressed"}

Don't import:

Import[#, "VideoEncoding"] & /@
 {"https://code.ros.org/trac/opencv/export/7213/trunk/opencv/samples/cpp/tutorial_code/HighGUI/video-input-psnr-ssim/video/Megamind.avi",
  "http://samples.mplayerhq.hu/avi/verona60avi56k.avi",
  "http://samples.mplayerhq.hu/avi/filedoesitbetter.avi",
  "http://www.infognition.com/ScreenPressor/browsing-divx.avi"
  }

{"XVID", "MP42", "MJPG", "DX50"}

Answer 1

64-bit Windows only

Note for Mathematica 11.3: There is a potential conflict between MathMF and the built-in MediaTools package. See here for details and here for an example of how to use MediaTools in place of MathMF.

 

Note for Mathematica version 10: The Wolfram Library has been updated in version 10 and you will need to recompile the MathMF DLL. This is most easily accomplished by evaluating "MathMF"//FindLibrary//DeleteFile prior to loading the package.

Link to package on GitHub

I have written a package called MathMF which uses a LibraryLink DLL to do frame-by-frame video import and export with Windows Media Foundation. It should be able to read a reasonable variety of movie files, including AVI, WMV and MP4. Exporting is currently limited to WMV and MP4 formats (AVI encoding is not natively supported by Media Foundation)

Here is the sort of code you can write with it. The code first opens a video file for reading, and creates a new video file for writing to. It then runs a loop in which each frame is sequentially read from the input stream, processed in Mathematica and then written to the output stream. So Mathematica is effectively being used as a video filter.

{duration, framerate, width, height} =
 MFInitSourceReader["C:\\Users\\Simon\\Desktop\\test1.wmv"];

MFInitSinkWriter["C:\\Users\\Simon\\Desktop\\filtered.wmv", 
 width, height, "FrameRate" -> framerate]

While[
  (image = MFGrabFrame["RealImage"]) =!= EndOfFile,
  MFSendFrame @ GradientFilter[image, 2]
  ] ~Monitor~ image

MFFinaliseSink[]

The package can be downloaded from the GitHub link at the top of this post, it is too large to include in full here.

The package includes the library source code, and on first use will attempt to compile the library locally. I believe the compilation should work if you have Visual Studio 2010 or later installed, and probably won't work if you use a different compiler. There is a pre-built DLL available if the compilation fails (see the readme on GitHub for more details)

I hope some people find this useful, it has been hovering in my mind as something to try to do for quite some time, hindered mainly by my total lack of experience with C++ and COM programming.

Answer 2

(Thanks to K.H. from Wolfram):

Even on Windows, if QuickTime is installed, Mathematica uses the native QuickTime install (excluding added, third-party codecs) to import AVI files, and the supported codecs are listed in:

Internal`$VideoEncodings

(Without an installation of QuickTime, only uncompressed AVI files are supported by Mathematica.) For the file http://people.sc.fsu.edu/~jburkardt/data/avi/ccvt_box.avi, QuickTime on Windows can not open it, so Mathematica has problem importing it.

Answer 3

If you only need to access the video frame by frame you can use ffmpeg to decode the video.

Reposting a previous answer:

---

If you only want to read it linearly you can tell ffmpeg to dump the video to stdout and then read widthheightbytes-per-pixel bytes at a time to get the video frame-by-frame:

openVideo[fname_, w_, h_] := 
 Module[{video}, 
  video["stream"] = 
   OpenRead[
    "!ffmpeg -i " ~~ fname ~~ " -loglevel quiet -f rawvideo -pix_fmt rgb24 -", 
    BinaryFormat -> True];
  video["SkipFrame", n_Integer: 1] := Skip[video["stream"], Byte, n*3*w*h];
  video["NextFrame"] := Partition[Partition[
     BinaryReadList[video["stream"], "Byte", 3*w*h]
   , 3], w];
  video["NextFrame", n_Integer] := Table[video["NextFrame"], {n}];
  video["NextImage"] := Image[video["NextFrame"], "Byte"];
  video["NextImage", n_Integer] := Table[video["NextImage"], {n}];
  video]

Here's an example:

(* Create a test movie *)
file = $TemporaryPrefix <> "testvid.avi";
Export[file,Table[Rasterize[i, ImageSize -> {352, 200}], {i, 1, 50}]];

video = openVideo[file,352,200]
video["NextImage",2]
video["SkipFrame",10]
video["NextImage"]

For some reason Mathematica seems to read through the entire video when doing Close[video["stream"]] and it might be worth to kill -15 the ffmpeg process manually to speed it up for large files.

For a 720x404 h264 mp4 video on a modest laptop it takes 13s to skip 200 frames and 0.07s to read a frame. The skipping could possibly be sped up by starting a new ffmpeg process that begins at desired frame. I would compare speeds to Import but Mathematica can't read it (for the test video Import takes .2s for first frame compared to .02s ). So as a bonus with this way you can work with many many more video formats :)

Answer 4

I think the problem is the fact that you cannot open the video at

http://people.sc.fsu.edu/~jburkardt/data/avi/ccvt_box.avi

by QuickTime. Try it. Download free QuickTime and try to open it in that player and is most probably won't work. It means that to make the video a non-standard video encoding was used which is not a subset of formats understood by QuickTime. High chance this is why M. cannot open it - my best guess.