Questions about RAM usage, including optimizing memory use, avoiding and fixing leaks and releasing unused memory.

learn more… | top users | synonyms

39
votes
9answers
2k views

What would you ask Mathematica to do on a big system?

Our hardware supplier has given us the opportunity to test Mathematica on a big Windows system. The specs will be 1 to 4 TB of working memory and 12 sockets with 18 cores each. I got 1 full week of ...
37
votes
4answers
2k views

Aborting evaluation when the memory exceeds a certain limit

Sometimes when generating a lot of data the memory usage shoots beyond its physical realm and my laptop freezes. There is no way to stop Mathematica when this happens. Task manager (or its equivalent ...
29
votes
3answers
2k views

Smooth Peter de Jong attractor

Today I was playing with Peter de Jong attractor. At the bottom of the page I've linked there are beautiful examples like: My attempts are not so great: It is around 10^5 points. For more than 5*...
29
votes
3answers
1k views

DeleteDuplicatesBy is not performing as I'd hoped. Am I missing something?

Déjà vu: a new-in-v10 function should provide a better solution to an old problem, but my enthusiasm is curbed when I run timings. This time the function is ...
29
votes
3answers
2k views

How to place an image inside a notebook, with the minimum memory footprint?

When creating a notebook with images, like photos, the file tends to quickly get big, which makes it difficult to use the notebook format for training material containing a lot of images. Are there ...
28
votes
4answers
3k views

Plotting large datasets

I have a lot of data (e.g., 64x8192), to plot and it consumes too much memory (more than 2 GB) causing Mathematica to return "no memory available". For example, ...
28
votes
2answers
2k views

Debugging memory leaks

I've written my incredibly complex, incredibly elegant analysis function, that works great on small test data. But when I run it on my real (bigger) data set it keeps running out of memory. It turns ...
26
votes
3answers
1k views

How to clear parts of a memoized function?

I have a function of two variables, e.g.: f[a_, b_] := f[a, b] = something f[a - 1, b - 1] etc With the above code I used the concept of memoization to speed up ...
26
votes
4answers
1k views

How to find out the (current) free memory on the system?

Is there any way in Mathematica to find out the currently free memory on the system Mathematica runs on (like the utility free shows on the command line)? I've ...
26
votes
1answer
3k views

Finding long strings of identical digits in transcendental numbers

Introduction Describing the three main streams of present-day mathematical philosophy (formalism, Platonism and intuitionism) in a well-known book, The Emperor's New Mind, R. Penrose says: ...it ...
25
votes
4answers
1k views

What is the fastest way to count square-free words?

Background A word is a string of letters in an alphabet. A square-free word has no adjacent repeating substring. For example, (in the ternary alphabet of {0,1,2}) the words 00, 012121, and 0212012021 ...
24
votes
1answer
712 views

Efficient lazy weak compositions

In Mathematica all weak k-compositions of integer n can be generated using permutations of integer partitions: ...
24
votes
1answer
822 views

Excessive memory use by Mathematica kernel in 10.0.2 after executing simple commands

Bug introduced in 10.0 and fixed in 10.1 Warning: the code below may hang your system if you have the Suggestions Bar enabled in Mathematica < 10.1. When issuing the following simple ...
22
votes
4answers
544 views

Running out of memory when partitioning an image with ImagePartition

I have an Image3D object of image type "Bit16" with the dimensions 1472 x 1472 x 1472 that I would like to split into small image cubes of dimension 64 x 64 x 64 ...
22
votes
3answers
2k views

Does parallel programming use up large quantities of memory in Mathematica?

When I start Mathematica with a fresh kernel and load a program I am developing to analyze biology data, the amount of used system memory is 1.9 GB (free memory is 14.5 GB). These memory values are ...
20
votes
3answers
714 views

Memory Leak in Frontend - anyone know a workaround?

When using Rasterize on a Graphics expression composed of many differently coloured Graphics objects memory is allocated in the frontend which is then not released or reused when a similar graphic ...
20
votes
2answers
912 views

Why does ImageData need four times more memory?

Background We are dealing with very large TIFF image files that are imported, processed and exported using Mathematica 9.0.1. Many of our algorithms only work with an array representation of the ...
17
votes
2answers
477 views

Module trash collection behaviour

I have asked this question before on the Wolfram community forum (http://community.wolfram.com/groups/-/m/t/83845?p_p_auth=Gxoxe65B) but not with great success. Module creates temporary variables of ...
17
votes
1answer
375 views

Unpacked eigenvectors of complex matrices

Edit: This bug is fixed in Mathematica 10.0.0 I want to calculate eigenvectors of a Hermitian matrix. For example ...
16
votes
2answers
3k views

How do you deal with very large datasets in Mathematica?

This question is in two related parts. The first is about dataset size, the second data wrangling. Part 1: I've built a naive Bayesian classifier on Mathematica. It uses ...
16
votes
1answer
360 views

Why does this line increase the consumed memory?

Probably this is a simple question, however I really don't understand this behavior in Mathematica. It particularly leads to a memory leak in a huge program I have to do processing of 3D images. I ...
14
votes
2answers
631 views

Is there any harm or benefit to Removing unneeded private symbols in packages?

One of the things I noticed is that in typical medium-to-large sized packages, a huge number of symbols are created in the myPackage`Private context. You can see a ...
14
votes
3answers
278 views

On making a graphical real-time memory usage monitor

I'd like to make a graphical version of Dynamic[Refresh[MemoryInUse[], UpdateInterval -> 1]] A naive implementation would be something like: ...
14
votes
1answer
740 views

Space-efficient null space of sparse array

I have a roughly 100,000 × 3,000 matrix (as a SparseArray) that I'd like to find the kernel (null space) of. It has about 500,000 nonzero entries, all -1 or 1. ...
14
votes
1answer
653 views

Efficient code for tallying entries in very large lists

I am dealing with data sets containing tens of millions of (hashable) entries and simply using the Tally function to count the frequency of each unique list element ...
14
votes
1answer
404 views

How to constrain memory usage on HPC?

I found the performance of built-in function MemoryConstrained is very poor. It stops the evaluation after the memory limit has already been exceeded greatly than ...
12
votes
1answer
194 views

ListConvolve Memory Hemorrhage

Bug introduced in 10.0.0 and fixed in 10.2.0. I wrote a routine that convolves an impulse response array with a test array and checks for the RMS difference between that convolution and a data ...
11
votes
2answers
737 views

Rendering thousands of points

I am trying to render an image with some 120,000 Points. Technically it's just 10,000 points, plus their reflection, plus 5 rotations of both of those sets, but I'm ...
11
votes
2answers
419 views

Giving JLink Huge Memory by Default

I have noticed that Mathematica makes use of Java by way of JLink, for example, in the extensions such as DatabaseLink and (I think) RLink. It also seems like Java is a first-class citizen for writing ...
11
votes
1answer
458 views

Nintegrate has memory leak

Bug introduced in 6.0 and fixed in 10.2.0 NIntegrates uses memory and does not release it. For long loops with NIntegrate ...
11
votes
1answer
323 views

Old values are not freed/garbage collected when you re-evaluate an assignment

For this code: (* Cell 1 *) generate := Module[{x}, x = Range[100 * 1000 * 1000]; x]; (* Cell 2 *) g = generate[]; MemoryInUse[] If I evaluate cell 2 repeatedly,...
11
votes
1answer
287 views

Where is my memory? — LibraryLink never returns the memory

Bug introduced in 8.0 or earlier and persisting through 10.3 This is an example LibaryLink code from Mathematica documentation, which takes an integer and return an integer list: ...
11
votes
2answers
871 views

Avoiding an unresponsive user interface in OS X

I have found that despite Mathematica's numerous updates, each of which have added much functionality, one fundamental issue remains unaddressed: The unresponsiveness of the UI when I make a mistake (...
11
votes
3answers
274 views

Can I return lists with different dimensions from a compiled function?

Sometimes I'll need several lists with different dimensions to be returned by a compiled function, but compiled function will fail if an irregular list contained: ...
11
votes
2answers
1k views

Why does this simple program leak memory?

I have a simple Mathematica program which writes some plots to image files for later conversion into a movie. Unfortunately, the program leaks so much memory that it quickly exhausts all 12G of RAM ...
11
votes
2answers
400 views

Memory leak in FE?

A very abridged example of what was originally a major leak ...
11
votes
1answer
221 views

How can the Theil-Sen estimator be made to work on larger datasets?

The Theil-Sen estimator finds the slope and intercept of a line passing through a set of points by calculating the median slope and median intercept of the set of lines passing through all possible ...
11
votes
0answers
202 views

Memory usage of compiled function: C vs WVM

Reported to WRI [CASE:3406803] Why does generating an array in "C"-compiled function uses 50% more memory than doing same thing in ...
10
votes
2answers
623 views

Memory leak issues

I am working with noncommuting objects and basically I am using Mathematica to sort large expressions by normal ordering and simplifying them. To this end, I first create an object called ...
10
votes
1answer
508 views

GeoDirection and GeoDistance Memory Leaks: How to Recover the Memory?

Bug introduced in 7.0 and fixed in 9.0.0 The functions GeoDirection and GeoDistance have memory leaks. To see this, run the example functions below and watch as the Physical Memory Available in ...
10
votes
2answers
262 views

Is there a way to control V10's undo length?

This may be a duplicate (if so I will remove this), but I was having trouble finding the relevant bit of documentation to fix it. I noticed that my FrontEnd memory ...
10
votes
1answer
99 views

How to speed up the evaluation of a function with a large LeafCount but many repeated sub-functions for use in NIntegrate

General Context Apparently, the memory leak in NIntegrate is unavoidable pre-MMA-10.2. I have some (two-dimensional) numerical integration to do on a complicated ...
10
votes
1answer
161 views

Memory leak with NDSolve

Edit The technical team confirms the memory leak in NDSolve and has forwarded an incident report to the developers. This is not the first time the question in ...
9
votes
2answers
454 views

How to un-eat memory?

Consider this code: ...
9
votes
2answers
298 views

Remove versus ClearAll

What is the difference between Remove and ClearAll? I thought that "not being recognized by Mathematica", which is the effect ...
9
votes
2answers
264 views

How can I tell how much memory is taken by an expression?

ByteCount and LeafCount don't really work, as shown below. Mathematica 9.0 on a 32-bit Windows machine: ...
9
votes
2answers
201 views

Comparing memory consumption of two pieces of code

I have implemented something I want to do in two different ways, because I'm not sure which of the two approaches give best speed/memory efficiency. Comparing the speed of the two approaches is ...
9
votes
1answer
1k views

Mathematica memory management for large arrays

I have come across a weird phenomenon in Mathematica when dealing with large arrays. When generating a list with all the possible subsets of three elements of another list (thus having elements which ...
9
votes
1answer
258 views

Is it possible to lower the bitwidth of (floating point) numbers?

The answer to this question will most probably be a definite no. Nevertheless, I would like to know whether it is possible to represent reals in the memory on less than the standard 32 or 64 bits (...
9
votes
0answers
446 views

How can I find the source of a memory leak?

I found a really weird issue using NDSolve`ProcessEquations inside module. We start by setting HistoryLength to 0. After this we define a function which takes in an integer and creates a Table of ...