Questions about RAM usage, including optimizing memory use, avoiding and fixing leaks and releasing unused memory.

learn more… | top users | synonyms

33
votes
4answers
2k views

Aborting evaluation when the memory exceeds a certain limit

Sometimes when generating a lot of data the memory usage shoots beyond its physical realm and my laptop freezes. There is no way to stop Mathematica when this happens. Task manager (or its equivalent ...
25
votes
2answers
2k views

Debugging memory leaks

I've written my incredibly complex, incredibly elegant analysis function, that works great on small test data. But when I run it on my real (bigger) data set it keeps running out of memory. It turns ...
22
votes
3answers
843 views

DeleteDuplicatesBy is not performing as I'd hoped. Am I missing something?

Déjà vu: a new-in-v10 function should provide a better solution to an old problem, but my enthusiasm is curbed when I run timings. This time the function is ...
21
votes
4answers
2k views

Plotting large datasets

I have a lot of data (e.g., 64x8192), to plot and it consumes too much memory (more than 2 GB) causing Mathematica to return "no memory available". For example, ...
23
votes
3answers
1k views

How to place an image inside a notebook, with the minimum memory footprint?

When creating a notebook with images, like photos, the file tends to quickly get big, which makes it difficult to use the notebook format for training material containing a lot of images. Are there ...
25
votes
4answers
1k views

How to find out the (current) free memory on the system?

Is there any way in Mathematica to find out the currently free memory on the system Mathematica runs on (like the utility free shows on the command line)? I've ...
17
votes
3answers
976 views

How to clear parts of a memoized function?

I have a function of two variables, e.g.: f[a_, b_] := f[a, b] = something f[a - 1, b - 1] etc With the above code I used the concept of memoization to speed up ...
18
votes
3answers
612 views

Memory Leak in Frontend - anyone know a workaround?

When using Rasterize on a Graphics expression composed of many differently coloured Graphics objects memory is allocated in the frontend which is then not released or reused when a similar graphic ...
3
votes
2answers
296 views

Storing result from nested Table in a File for further computations

Consider the following: Table[Table[x = 20; y = 30; 2 i x + 3 j y , {i, 10}], {j, 3}] Suppose I want the result of this nested Table to be stored to an ...
8
votes
1answer
199 views

Is it possible to lower the bitwidth of (floating point) numbers?

The answer to this question will most probably be a definite no. Nevertheless, I would like to know whether it is possible to represent reals in the memory on less than the standard 32 or 64 bits ...
5
votes
1answer
98 views

How to relate memory usage with occupied positions of SparseArrays?

What is the relation of memory usage of a SparseArray and the number of its occupied positions? Let's say you build a 100.000.000 by 10 element SparseArray. And fill the two position 1/1 and ...
15
votes
2answers
2k views

How do you deal with very large datasets in Mathematica?

This question is in two related parts. The first is about dataset size, the second data wrangling. Part 1: I've built a naive Bayesian classifier on Mathematica. It uses ...
25
votes
1answer
3k views

Finding long strings of identical digits in transcendental numbers

Introduction Describing the three main streams of present-day mathematical philosophy (formalism, Platonism and intuitionism) in a well-known book, The Emperor's New Mind, R. Penrose says: ...it ...
22
votes
1answer
566 views

Excessive memory use by Mathematica kernel in 10.0.2 after executing simple commands

Bug introduced in 10.0 and fixed in 10.1 Warning: the code below may hang your system if you have the Suggestions Bar enabled in Mathematica < 10.1. When issuing the following simple ...
7
votes
2answers
267 views

Clean up list of partial outputs for memory

Context I run long Mathematica sessions; sometimes I have Output up to Out[3000+]. Hence I would like to be able to clear out some previous outputs which do use lots of memory. Attempt I am aware ...
5
votes
2answers
450 views

Memory efficient generation and selection of tuples

The short of this question: I need to find a bunch of n-tuples of {0,1} which do not fail to satisfy a set of (non-linear polynomial) equations, without hogging up all of my memory by trying to ...
11
votes
1answer
610 views

Space-efficient null space of sparse array

I have a roughly 100,000 × 3,000 matrix (as a SparseArray) that I'd like to find the kernel (null space) of. It has about 500,000 nonzero entries, all -1 or 1. ...
4
votes
3answers
235 views

Is there something wrong with this piece of code involving Manipulate?

I am playing around with the Manipulate sentence. I think the following code is right, and actually Mathematica 9.0 doesn't return any error message, but I am ...
26
votes
3answers
1k views

Smooth Peter de Jong attractor

Today I was playing with Peter de Jong attractor. At the bottom of the page I've linked there are beautiful examples like: My attempts are not so great: It is around 10^5 points. For more than ...
7
votes
1answer
785 views

Memory not freed after running Clear when using a Table

When I'm creating large lists with Table I see the memory footprint increasing. ...
11
votes
2answers
359 views

Memory leak in FE?

A very abridged example of what was originally a major leak ...
7
votes
3answers
704 views

How to fix this memory leak

I am using halirutan's code in this thread to unflatten a list. It works very well but the problem is that the Module's local function f is not deleted when the ...
11
votes
1answer
247 views

Where is my memory? — LibraryLink never returns the memory

Bug introduced in 8.0 or earlier and persisting through 10.2.0 This is an example LibaryLink code from Mathematica documentation, which takes an integer and return an integer list: ...
11
votes
2answers
869 views

Why does this simple program leak memory?

I have a simple Mathematica program which writes some plots to image files for later conversion into a movie. Unfortunately, the program leaks so much memory that it quickly exhausts all 12G of RAM ...
9
votes
1answer
385 views

Nintegrate has memory leak

Bug introduced in 6.0 and fixed in 10.2.0 NIntegrates uses memory and does not release it. For long loops with NIntegrate ...
6
votes
2answers
100 views

Possible memory leak when importing JSON

Bug introduced in 10.0 and persisting though 10.2.0 This problem that I would like to resolve is how to free memory used Import a file when the file is no longer in use. I have a project in which ...
22
votes
3answers
1k views

Does parallel programming use up large quantities of memory in Mathematica?

When I start Mathematica with a fresh kernel and load a program I am developing to analyze biology data, the amount of used system memory is 1.9 GB (free memory is 14.5 GB). These memory values are ...
19
votes
2answers
800 views

Why does ImageData need four times more memory?

Background We are dealing with very large TIFF image files that are imported, processed and exported using Mathematica 9.0.1. Many of our algorithms only work with an array representation of the ...
14
votes
1answer
584 views

Efficient code for tallying entries in very large lists

I am dealing with data sets containing tens of millions of (hashable) entries and simply using the Tally function to count the frequency of each unique list element ...
16
votes
1answer
329 views

Unpacked eigenvectors of complex matrices

Edit: This bug is fixed in Mathematica 10.0.0 I want to calculate eigenvectors of a Hermitian matrix. For example ...
11
votes
1answer
292 views

Old values are not freed/garbage collected when you re-evaluate an assignment

For this code: (* Cell 1 *) generate := Module[{x}, x = Range[100 * 1000 * 1000]; x]; (* Cell 2 *) g = generate[]; MemoryInUse[] If I evaluate cell 2 ...
10
votes
2answers
244 views

Is there a way to control V10's undo length?

This may be a duplicate (if so I will remove this), but I was having trouble finding the relevant bit of documentation to fix it. I noticed that my FrontEnd memory ...
9
votes
2answers
573 views

Memory leak issues

I am working with noncommuting objects and basically I am using Mathematica to sort large expressions by normal ordering and simplifying them. To this end, I first create an object called ...
8
votes
2answers
343 views

How to un-eat memory?

Consider this code: ...
10
votes
2answers
282 views

Giving JLink Huge Memory by Default

I have noticed that Mathematica makes use of Java by way of JLink, for example, in the extensions such as DatabaseLink and (I think) RLink. It also seems like Java is a first-class citizen for writing ...
8
votes
2answers
245 views

How can I tell how much memory is taken by an expression?

ByteCount and LeafCount don't really work, as shown below. Mathematica 9.0 on a 32-bit Windows machine: ...
8
votes
1answer
758 views

Mathematica memory management for large arrays

I have come across a weird phenomenon in Mathematica when dealing with large arrays. When generating a list with all the possible subsets of three elements of another list (thus having elements which ...
5
votes
3answers
456 views

How do I find the amount of free space on disk?

I know a workaround, where one calculates sizes of all files and then you subtract this from your default space, but is there any direct way to do it? Simplicity and speed are important. There's the ...
8
votes
1answer
153 views

AndersonDarlingTest and DistributionFitTest Leaking Memory

I believe AndersonDarlingTest and probably all other tests are keeping a copy of data somewhere and not freeing it: ...
8
votes
2answers
744 views

Working with large data, pass-by-reference, compiled functions

I think I know the answer to this, but after two days of Googling, this is my last effort before going down an unwanted path... I have a function that takes lots of large data as arguments as well as ...
4
votes
1answer
114 views

Memory usage of timeseries - lists of {Integer32, Real32}

A list of {Integer32, Real32} items in Mathematica seems to use 12-times as much memory than the C equivalent array of structs. ...
2
votes
1answer
96 views

Memory usage optimization for a large Association/SparseArray

Reformulating my question in simpler terms. Take two lists list1 = RandomInteger[{1, 100000000}, 25000000]; list2 = RandomReal[{1, 10}, 25000000]; Make an ...
2
votes
3answers
146 views

Accessing temporary value of a Nest evaluation

The function Nest[] must store the temporary values in the memory somewhere (those values that would form the entries of the list which is returned by the ...
6
votes
1answer
900 views

Clear and Unset doesn't free memory

Example: Try this example only if you have > 4G ram d = 2000; f[x_, y_, d_] = Sum[Subscript[a, i, j]*x^i*y^j, {i, 0, d - 1}, {j, 0, d - i}]; Then ...
5
votes
1answer
103 views

memory leaks using NIntegrate on parallel kernel

Bug fixed in version 10.2.0 I'm running a large computation in parallel and the memory usage of the parallel kernels is increasing with every iteration of the calculation. Neither ...
5
votes
3answers
247 views

Huge matrices crashes in Mathematica

I have a matrix of data ~(70,000x70,000). I am searching connection between these data points. I use euclidean distance for that purpose. If it is less than my threshold (that means they have ...
4
votes
1answer
130 views

Memory leak in MemberQ

Bug introduced in 10.0.0 and fixed in 10.0.2 Consider the following code. ...
3
votes
2answers
105 views

Memory Problem with Modules and NDSolve

I'm using NDSolve inside a module, and I appear to have a memory leak. The relevant code is: ...
3
votes
0answers
133 views

Increasing MaxPoints in NDSolve results in memory issue

I am interested in increasing "MaxPoints" in NDSolve's "MethodOfLines" in attempt to increase the resolution of the plot of the solution of linearly damped wave equation with transparent boundary ...
3
votes
1answer
142 views

May I restrict caching to disk when memory runs low? [duplicate]

An hour ago, I had to hard reset my laptop again because Mathematica froze the system again. When import a large file or make a mistake that produces huge arrays with complicated entries etc., I often ...