Questions about RAM usage, including optimizing memory use, avoiding and fixing leaks and releasing unused memory.

learn more… | top users | synonyms

25
votes
4answers
1k views

Aborting evaluation when the memory exceeds a certain limit

Sometimes when generating a lot of data the memory usage shoots beyond its physical realm and my laptop freezes. There is no way to stop Mathematica when this happens. Task manager (or its equivalent ...
23
votes
4answers
1k views

What is the fastest way to count square-free words?

Background A word is a string of letters in an alphabet. A square-free word has no adjacent repeating substring. For example, (in the ternary alphabet of {0,1,2}) the words 00, 012121, and 0212012021 ...
23
votes
4answers
874 views

How to find out the (current) free memory on the system?

Is there any way in Mathematica to find out the currently free memory on the system Mathematica runs on (like the utility free shows on the command line)? I've ...
23
votes
1answer
3k views

Finding long strings of identical digits in transcendental numbers

Introduction Describing the three main streams of present-day mathematical philosophy (formalism, Platonism and intuitionism) in a well-known book, The Emperor's New Mind, R. Penrose says: ...it ...
22
votes
3answers
867 views

How to place an image inside a notebook, with the minimum memory footprint?

When creating a notebook with images, like photos, the file tends to quickly get big, which makes it difficult to use the notebook format for training material containing a lot of images. Are there ...
20
votes
2answers
1k views

Debugging memory leaks

I've written my incredibly complex, incredibly elegant analysis function, that works great on small test data. But when I run it on my real (bigger) data set it keeps running out of memory. It turns ...
19
votes
3answers
931 views

Does parallel programming use up large quantities of memory in Mathematica?

When I start Mathematica with a fresh kernel and load a program I am developing to analyze biology data, the amount of used system memory is 1.9 GB (free memory is 14.5 GB). These memory values are ...
19
votes
2answers
575 views

Why does ImageData need four times more memory?

Background We are dealing with very large TIFF image files that are imported, processed and exported using Mathematica 9.0.1. Many of our algorithms only work with an array representation of the ...
18
votes
4answers
363 views

Running out of memory when partitioning an image with ImagePartition

I have an Image3D object of image type "Bit16" with the dimensions 1472 x 1472 x 1472 that I would like to split into small image cubes of dimension 64 x 64 x 64 ...
17
votes
4answers
1k views

Plotting large datasets

I have a lot of data (e.g., 64x8192), to plot and it consumes too much memory (more than 2 GB) causing Mathematica to return "no memory available". For example, ...
16
votes
3answers
499 views

How to clear parts of a memoized function?

I have a function of two variables, e.g.: f[a_, b_] := f[a, b] = something f[a - 1, b - 1] etc With the above code I used the concept of memoization to speed up ...
16
votes
3answers
428 views

Memory Leak in Frontend - anyone know a workaround?

When using Rasterize on a Graphics expression composed of many differently coloured Graphics objects memory is allocated in the frontend which is then not released or reused when a similar graphic ...
14
votes
1answer
258 views

How to constrain memory usage on HPC?

I found the performance of built-in function MemoryConstrained is very poor. It stops the evaluation after the memory limit has already been exceeded greatly than ...
13
votes
2answers
181 views

Module trash collection behaviour

I have asked this question before on the Wolfram community forum (http://community.wolfram.com/groups/-/m/t/83845?p_p_auth=Gxoxe65B) but not with great success. Module creates temporary variables of ...
13
votes
1answer
422 views

Efficient code for tallying entries in very large lists

I am dealing with data sets containing tens of millions of (hashable) entries and simply using the Tally function to count the frequency of each unique list element ...
12
votes
1answer
205 views

Unpacked eigenvectors of complex matrices

I want to calculate eigenvectors of a Hermitian matrix. For example ...
11
votes
1answer
436 views

Space-efficient null space of sparse array

I have a roughly 100,000 × 3,000 matrix (as a SparseArray) that I'd like to find the kernel (null space) of. It has about 500,000 nonzero entries, all -1 or 1. ...
11
votes
2answers
928 views

How do you deal with very large datasets in Mathematica?

This question is in two related parts. The first is about dataset size, the second data wrangling. Part 1: I've built a naive Bayesian classifier on Mathematica. It uses ...
11
votes
2answers
280 views

Memory leak in FE?

A very abridged example of what was originally a major leak ...
10
votes
1answer
208 views

Old values are not freed/garbage collected when you re-evaluate an assignment

For this code: (* Cell 1 *) generate := Module[{x}, x = Range[100 * 1000 * 1000]; x]; (* Cell 2 *) g = generate[]; MemoryInUse[] If I evaluate cell 2 ...
10
votes
3answers
218 views

Can I return lists with different dimensions from a compiled function?

Sometimes I'll need several lists with different dimensions to be returned by a compiled function, but compiled function will fail if an irregular list contained: ...
9
votes
2answers
583 views

Avoiding an unresponsive user interface in OS X

I have found that despite Mathematica's numerous updates, each of which have added much functionality, one fundamental issue remains unaddressed: The unresponsiveness of the UI when I make a mistake ...
9
votes
2answers
462 views

Memory leak issues

I am working with noncommuting objects and basically I am using Mathematica to sort large expressions by normal ordering and simplifying them. To this end, I first create an object called ...
9
votes
1answer
433 views

GeoDirection and GeoDistance Memory Leaks: How to Recover the Memory?

The functions GeoDirection and GeoDistance have memory leaks. To see this, run the example functions below and watch as the Physical Memory Available in the Windows Task Manager, Performance tab, ...
9
votes
2answers
536 views

Why does this simple program leak memory?

I have a simple Mathematica program which writes some plots to image files for later conversion into a movie. Unfortunately, the program leaks so much memory that it quickly exhausts all 12G of RAM ...
8
votes
1answer
175 views

Reducing the memory footprint of lists of machine numbers

I have a large 1D dataset of double precision numbers that are stored in many external database files. In order to load the whole dataset into Mathematica 8 I iterate over the files ...
8
votes
1answer
166 views

Where is my memory? — LibraryLink never returns the memory

This is an example LibaryLink code from Mathematica documentation, which takes an integer and return an integer list: ...
8
votes
1answer
132 views

AndersonDarlingTest and DistributionFitTest Leaking Memory

I believe AndersonDarlingTest and probably all other tests are keeping a copy of data somewhere and not freeing it: ...
7
votes
2answers
209 views

Clean up list of partial outputs for memory

Context I run long Mathematica sessions; sometimes I have Output up to Out[3000+]. Hence I would like to be able to clear out some previous outputs which do use lots of memory. Attempt I am aware ...
7
votes
3answers
400 views

How to fix this memory leak

I am using halirutan's code in this thread to unflatten a list. It works very well but the problem is that the Module's local function f is not deleted when the ...
7
votes
2answers
588 views

Working with large data, pass-by-reference, compiled functions

I think I know the answer to this, but after two days of Googling, this is my last effort before going down an unwanted path... I have a function that takes lots of large data as arguments as well as ...
7
votes
2answers
653 views

Memory running out generating ContourPlot3D

The code below ...
7
votes
1answer
611 views

Memory not freed after running Clear when using a Table

When I'm creating large lists with Table I see the memory footprint increasing. ...
7
votes
2answers
163 views

How to un-eat memory?

Consider this code: ...
7
votes
1answer
400 views

Mathematica memory management for large arrays

I have come across a weird phenomenon in Mathematica when dealing with large arrays. When generating a list with all the possible subsets of three elements of another list (thus having elements which ...
6
votes
2answers
140 views

Giving JLink Huge Memory by Default

I have noticed that Mathematica makes use of Java by way of JLink, for example, in the extensions such as DatabaseLink and (I think) RLink. It also seems like Java is a first-class citizen for writing ...
6
votes
2answers
282 views

Memory management and speed for Fast GCD

Let's say that we have some $300\,\text{K}$ digits (arbitrary function) and want to trial factor with $100{,}000{,}000$ first prime numbers. ...
6
votes
1answer
110 views

Is there a version of Together that doesn't cancel common factors?

I'm running code that produces a long combination of rational functions with free constants that need to be tuned such that the combination vanishes. My current method is to put the combination over ...
5
votes
2answers
134 views

Remove versus ClearAll

What is the difference between Remove and ClearAll? I thought that "not being recognized by Mathematica", which is the effect ...
5
votes
1answer
65 views

How to relate memory usage with occupied positions of SparseArrays?

What is the relation of memory usage of a SparseArray and the number of its occupied positions? Let's say you build a 100.000.000 by 10 element SparseArray. And fill the two position 1/1 and ...
5
votes
1answer
192 views

Clear and Unset doesn't free memory

Example: Try this example only if you have > 4G ram d = 2000; f[x_, y_, d_] = Sum[Subscript[a, i, j]*x^i*y^j, {i, 0, d - 1}, {j, 0, d - i}]; Then ...
5
votes
1answer
216 views

Processing and storing a graph with a large number of vertices s.t. k-hop neighborhoods are efficiently retrievable

I have an unweighted, undirected graph $G$ with $(v_1, ..., v_N) \in V$ vertices, where each vertex in the lattice represents a string of some length ($\leq 200$ characters), and I need to construct ...
5
votes
1answer
77 views

Does Share have any side effect?

According to the documentation, Share[expr] changes the way expr is stored internally, to try and minimize the amount of ...
5
votes
0answers
147 views

How can I find the source of a memory leak?

I found a really weird issue using NDSolve`ProcessEquations inside module. We start by setting HistoryLength to 0. After this we define a function which takes in an integer and creates a Table of ...
5
votes
0answers
159 views

NDSolve and memory usage

After some googling, i've found similar problems around, but didn't find a 100% satisfactory answer, so let me ask here: I'd like to solve a 1+1 problem using the method of lines. In spherical ...
5
votes
1answer
117 views

Is it possible to lower the bitwidth of (floating point) numbers?

The answer to this question will most probably be a definite no. Nevertheless, I would like to know whether it is possible to represent reals in the memory on less than the standard 32 or 64 bits ...
4
votes
2answers
273 views

Implementation of builtin function Fold

In the documentation for fold, it says Fold[f,x,list] gives the last element of FoldList[f,x,list]. But this is surely not the way it is implemented, right? ...
4
votes
2answers
245 views

Memory efficient generation and selection of tuples

The short of this question: I need to find a bunch of n-tuples of {0,1} which do not fail to satisfy a set of (non-linear polynomial) equations, without hogging up all of my memory by trying to ...
4
votes
3answers
305 views

How do I find the amount of free space on disk?

I know a workaround, where one calculates sizes of all files and then you subtract this from your default space, but is there any direct way to do it? Simplicity and speed are important. There's the ...