Memory problem in exporting images

Question

I saw discussions of similar questions in the past, still, I will ask mine. It seems there is a memory leak in exporting images with recent versions of Mathematica - 13 and 14.

What happens is the following. I am doing a Monte Carlo simulation and generating frames for a video. Frames are stored in the list of frames. Surprisingly, although there are thousands of frames (5000-8000), accumulating them does not lead to a large memory increase, and I feel comfortable having 32 GB RAM on a Windows 11 PC.

However, problems begin when I start making a video of these frames. Direct exporting them into a video with the command Export[....mp4] or similar gives an error after a long number crunching. I've got an idea to first export the frames as images in a directory and then make a video from them with ffmpeg. The latter works very fast and makes no problems. The problem is with Mathematica.

Exporting frames as images one by one in a Do cycle (I tried PNG and TIF) results in a steady increase in the memory used, according to the Windows Task Manager. This was totally unexpected. I thought that exporting the next image would simply use the same memory that was used by the preceding one. In reality, all my 32 GB of RAM are getting slowly filled, and then there is a memory-release process: Memory usage goes up and down a little bit below 100%. I cannot launch other applications like a web browser since there is no memory for them.

This goes on for some time - I can get about 2000 images exported, then finally there is a memory exception and the kernel quits losing all data, or Mathematica itself quits. I noticed that the problem with memory is more severe in Mathematica 14 than in Mathematica 13. After these crashes and losing the computed data, I decided to save the list of frames on the disk as a Mathematica expression *.m before exporting images. So now the workflow is like this: I am exporting images until Mathematica crashes, then I import the saved frames and continue exporting where it stopped. Like this, I can export about 7500 images in three rounds. Then I make videos of about 1 GB size with ffmpeg without problems.

Just now the kernel quit after exporting over a thousand frames ("Not enough memory to rasterize images") but the memory is not released. So I need to close Mathematica to release memory and then continue.

Maybe somebody could suggest a better workflow for making videos? In this group, I have seen a post where images were exported with something having <> instead of the Export command. Is it the same or better?

I can rasterize frames before saving them - is it helpful? Or rasterization of many images will overfill the memory?

I also have a Mac Pro computer and a Linux workstation at work. Could it be better to make videos on them?

In 2022 I was making similar videos with a previous version of Mathematica - 11 or 12. Back then, I was able to export videos directly by Mathematica's command Export[...mp4] or Export[...flv], and Mathematica didn't tell me to install FFmpeg as it is doing now.

Answer 1

I have found a quasi-answer to this question. As rasterization of images is a very heavy procedure with a memory leak, the best way to proceed is to

1) Compute the list of frames;
2) Save it to the disk as a *.m file;
3) Exit Mathematica;
4) Use Wolframscript to read the *.m file from the disk and rasterize images exporting them numbered on the disk one-by-one.
5) Use ffmpeg to make a video from the exported images.

Exporting a big video directly from Mathematica by the Export command is likely to fail after a long number crunching. Rasterization of the frames in Mathematica does not work well because of the memory leak. As RAM becomes full, Mathematica's front end starts to freeze saying "Formatting Notebook content".

It turns out that Woframscript calls Mathematica which works in the background, according to the Windows Task Manager. They work together rasterizing frames and the memory usage slowly grows because of the memory leak. RAM becomes full and after some time rasterizing slows down. Here comes an important know-how: You kill Mathematica in the Task Manager ("End the task"). As a result, the memory empties and rasterization goes fast again. Wolframscript evokes Mathematica again the process continues. When RAM becomes full again you repeat the procedure. Killing Mathematica results in one frame being sometimes corrupted or missing. However, if there are thousands of frames, this is not a big deal.

Of course, all this sounds funny when Mathematica is being advertised as a universal tool for everything and an ultimate enterprise solution. Its reliability is hardly at the level needed to satisfy companies. But as a research tool, it is still quite good.