1
$\begingroup$

I have a notebook that I do my calculations in. I can't really copy it here because it's both pretty big, and I can't share the content. But, when I save it, the process will often take several minutes... it stops responding and does the "window not responding" thing Windows does when it thinks it's frozen, though it eventually does finish saving.

I'm guessing it's because I have some fairly large data sets (matrices of dimensions 200x200x5) and also have them plotted (maybe that's the real problem?).

If I do:

In[128]:= N@MemoryInUse[]/10^6
N@MaxMemoryUsed[]/10^6

Out[128]= 55.4787

Out[129]= 74.8118

That doesn't seem that large...

If I look in Windows Task manager, Mathematica is currently using 45MB memory. If I look at the .nb file in Windows Explorer, it's 122MB large.

I've read about setting $HistoryLength=1, so I've done that just now. (Just to be clear, that shouldn't affect variables, right? Just the output that gets automatically saved so you can access it with % ?)

Is there anything else I'm missing? For example, would it make sense to, instead of plotting 40,000 points/graphics objects on a plot, raster that plot as an image, and ditch the data or something? It's just very frustrating having MMa constantly lag, hang, and freeze.

edit:

Let me include a more concrete example. I start off by doing $HistoryLength=1, by the way. Let's say I want to produce this horrible looking image:

nSquares = 200;
sqArray = 
  Flatten[#, 1] &@
   Table[Table[{XYZColor[{Sin[.1 i] i^.8, i*j/nSquares, 
         Sin[.1 j] j^.9}/nSquares], 
      Rectangle[{i, j}, {i + 1, j + 1}]}, {i, nSquares}], {j, 
     nSquares}];
Graphics[sqArray, ImageSize -> 800]

enter image description here

I don't think I can downsample it because it's 800 pixels wide, and there are only 200 color squares in each direction... I'd like high resolution for these images and I can already see granularity in this one.

What's strange is that after plotting this, MemoryInUse[] tells me only 47MB is being used, but Windows Task Manager says MMa is using 597MB... In fact, even after doing a Quit[] (which I thought would delete anything using memory?), task manager still says the same... So is this just from drawing that Graphics with the ~400k rectangles?

If so, what can I do here? I can certainly import pictures with better resolution that this easily... is the smart thing to Export[] it, then import it?

edit2: So it's definitely due to drawing the Graphics[]... If I put a semicolon so it doesn't draw it, the memory in Task Manager doesn't go high and stay there. But then I obviously want to see the image, so I tried the answer I got in this post, by doing:

testgraphics = Graphics[sqArray, ImageSize -> 800];
rast = Rasterize[#, "Image", RasterSize -> 300, ImageSize -> 600] &@
  testgraphics

And besides taking like... a full minute to Rasterize it (does that seem right?), it then takes 360MB in Task Manager that doesn't go away after doing Quit[] (similar to when displaying the Graphics).

So this is confusing still. Clearly just having an image in the notebook doesn't take much memory (I just imported one to test that's at a much higher resolution, and it didn't add any noticeable memory usage). This is significantly slowing down my workflow... what can I do? Thank you.

one more thing: I tried just exporting it, so that it never even displays in my notebook. It STILL increased the memory MMA is using in WTM to 350MB (after it's done exporting).

$\endgroup$
  • $\begingroup$ I suspect writing out your data plots is big part of the time it is tanking to save your notebook. I advise down-sampling the data before plotting. You cant't possibly distinguish 40,000 points on one plot at any reasonable image size. I doubt if your plot would look much different with 10000 points. And you might be able to get away with much less. $\endgroup$ – m_goldberg Jul 12 '17 at 23:43
  • $\begingroup$ @m_goldberg thanks for the response. I edited my question to make my problem a little more clear, I think you're right about where the memory/time is going, but I can't downsample in this case. $\endgroup$ – YungHummmma Jul 13 '17 at 1:30
  • $\begingroup$ MemoryInUse[] only gives kernel memory. You need to check MemoryInUse[$FrontEnd] as well. $\endgroup$ – m_goldberg Jul 13 '17 at 6:22
  • $\begingroup$ $HistoryLength will have no effect on the notebook size. As a rule of thumb: what-you-see-is-what-you-save. Therefore the less output cells, plot points, graphics elements, graphs, images, etc. the smaller the final size. A cool trick is to replace the output with an image whenever that is smaller. As in rasterAsNeeded[expr_] := With[{raster = Rasterize[expr, "Image"]}, If[ByteCount[raster] < ByteCount[expr], raster, expr]] $\endgroup$ – Batracos Jul 13 '17 at 19:04
  • $\begingroup$ @m_goldberg thank you, good point! that seems to be where all that memory is going... So here's my question: I made it so nothing actually gets displayed, that is, all my images/plots/etc, I have a semicolon at the end of the line, so it doesn't display, and I just make it export to a file. So nowhere in the actual notebook is the plot, but my memory is still high. What command can I do to release that memory? I tried Clear[] on the variable of the plot, but it didn't do it. $\endgroup$ – YungHummmma Jul 17 '17 at 3:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.