8
$\begingroup$

I will get large RGB AVI files (35 Hz, 1600$\times$1200 pixels) of up to 100 GB size. All three channels contain the same grayscale image. I want to extract each AVI file into one channel png images. To do that I wanted to use the following code snippet:

nImages = Length@Import[aviFileName, "Frames"];

ParallelDo
[
  image = ColorConvert[Import[aviFileName, {"Frames", i}], "Grayscale"]; 
  imageName = ToString@PaddedForm[i, 6, NumberPadding -> {"0", ""}];
  Image`ImportExportDump`ImageWritePNG[StringJoin[imageName, ".png"], image];
  , {i, 1, nImages}
];

Questions:

  1. Can such large files be read this way?

  2. How can the performance be improved?

$\endgroup$
  • 4
    $\begingroup$ This is not a very backed up answer, but from my experience it can be tough to work with files that big in Mathematica, because Mathematica usually tries to load everything into memory when Importing. Because of that i try to avoid loading files bigger than ˜1GB into Mathematica. For your specific problem i would try a specialised tool like VirtualDub or Adobe Premiere or FinalCutPro/Compressor if you're on a mac. But i would very much like if someone with more Mathematica knowledge proves me wrong :) $\endgroup$ – Thies Heidecke Mar 11 '17 at 15:53
  • 1
    $\begingroup$ Have you tried this on your avi file? I'm sure that importing all frames of 100GB avi files, which you do in your first line, won't work because it will load them into your RAM. If you have more than 100GB RAM indeed and this works, then one way to try would be to turn it into a list {{1, frame1},{2, frame2},...} and process this in parallel. Still, it is not sure that this will work. $\endgroup$ – halirutan Mar 11 '17 at 16:04
  • 1
    $\begingroup$ @mrz: ImageMagick also seems worth a look. It seems it could do both steps you want (avi->png and rgb->grayscale). $\endgroup$ – Thies Heidecke Mar 11 '17 at 16:07
  • 1
    $\begingroup$ If you are on Windows you can use MathMF from here. It reads frames one at a time so there should be no problem working with a very large file. $\endgroup$ – Simon Woods Mar 11 '17 at 17:31
  • 1
    $\begingroup$ ffmpeg would be my choice. $\endgroup$ – Felix Mar 12 '17 at 21:12
12
$\begingroup$

Here is an update where I compared the image extraction (8bit, 1 channel) from two AVI files:

enter image description here

My computer (Mathematica 11.0.1, Win 10 Pro, Intel i7 7490-MX, 3.1GHz, 4 cores) has 16 GB RAM and it never used in total more than 5 GB RAM, all running processes included.

The code can be used for multiple AVIs in different subdirectories in a certain base directory:

(* Read in folder which is base dir to search for avi files in all subdirs *)

ChoiceDialog[{FileNameSetter[Dynamic[baseDir], "Directory"], Dynamic[baseDir]}];

SetDirectory[baseDir];

aviNames = FileNames["*.avi", baseDir, Infinity];

numberAVIFiles = Length[aviNames];

aviDirs = DirectoryName[#] & /@ aviNames;

pngBaseDirs = FileBaseName[#] & /@ aviNames;

pngFullDirs = Transpose@{aviDirs, pngBaseDirs} /. {a_, p_} :> StringJoin[a, p];


(* Create a new directory for pngs if it does not exist otherwise delete it and create a new one *)
(If[DirectoryQ[#], DeleteDirectory[#, DeleteContents -> True]; CreateDirectory[#], CreateDirectory[#]]) & /@ 
  pngFullDirs;

Do[
   numberImages = Length@Import[aviNames[[j]], "Frames"];

   ParallelDo[

    image = ColorConvert[Import[StringJoin[aviNames[[j]]], {"Frames", i}], "Grayscale"];

    fileNameCounter = ToString@PaddedForm[i, 6, NumberPadding -> {"0", ""}];

    Image`ImportExportDump`ImageWritePNG[
     StringJoin[pngFullDirs[[j]], "\\", pngBaseDirs[[j]], "_", fileNameCounter, ".png"], image];

    fileName = StringJoin[pngBaseDirs[[j]], "_", fileNameCounter, ".png"];


    , {i, 1, numberImages}
    ];

   , {j, 1, numberAVIFiles}
   ];
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.