Questions about RAM usage, including optimizing memory use, avoiding and fixing leaks and releasing unused memory.

learn more… | top users | synonyms

3
votes
0answers
58 views

Memory Management for Large Datasets [on hold]

I've written some Mathematica code to generate polynomial roots. The code take an argument n as the highest degree of polynomial to solve for and then exports a file containing a list of the roots: ...
1
vote
0answers
27 views

Imported variables (.mx) are emptied occassionally (Linux, v10.0.2)

I have a severe problem with variables being "emptied" occassionally. I am not able to reproduce the behavior but this is basically what happens: 1) store list ...
11
votes
1answer
150 views

ListConvolve Memory Hemorrhage

Bug introduced in 10.0.0 and fixed in 10.2.0. I wrote a routine that convolves an impulse response array with a test array and checks for the RMS difference between that convolution and a data ...
11
votes
2answers
772 views

Avoiding an unresponsive user interface in OS X

I have found that despite Mathematica's numerous updates, each of which have added much functionality, one fundamental issue remains unaddressed: The unresponsiveness of the UI when I make a mistake ...
3
votes
1answer
55 views

Add Results of ParallelMap in Parallel

I'm new to coding in parallel, but I really could use it. I need a command to call a function on each element in a huge list, and then add them all up. Currently I use something like this: ...
4
votes
1answer
83 views

memory leaks using NIntegrate on parallel kernel

Bug fixed in version 10.2.0 I'm running a large computation in parallel and the memory usage of the parallel kernels is increasing with every iteration of the calculation. Neither ...
3
votes
1answer
82 views

Simplify a huge expression with limited memory

I would like to perform some analytical sums such as the following ...
5
votes
3answers
240 views

Huge matrices crashes in Mathematica

I have a matrix of data ~(70,000x70,000). I am searching connection between these data points. I use euclidean distance for that purpose. If it is less than my threshold (that means they have ...
2
votes
0answers
62 views

MatrixRank of a large but sparse matrix

I want to find the rank of a very sparse, quite rectangular matrix mat, but I'm running out of RAM (I have 16 GB) if I try to use ...
6
votes
0answers
131 views

NMinimize with NIntegrate (crash in symbolic evaluation, memory leak)

This is a "common" problem from what I've seen, but with a different spin. I have a function I use often that finds a fit of an expression to another expression with some free parameters (e.g. for ...
0
votes
0answers
32 views

Is it possible to change default $HistoryLength [duplicate]

By default $HistoryLength is set to \[Infinity]. This is often unnecessary and can cause unecessary memory usage. I can of ...
1
vote
0answers
45 views

Does Sum calculate all terms before summing them?

After calculating the sum of a large number of large objects, using B = Sum[RandomInteger[{0, 1}, {10^6}], {100}]; I find that ...
0
votes
2answers
40 views

Notebook cells space on disk profiler

I have one single notebook with more than 500 mb... To control its growth, I would like to have a profiler of used space (not the live memory, but the space on disk), in such a way that I can easily ...
2
votes
0answers
68 views

Pros/Cons to updating in place

There are often times when I have to update a list. The following is an example: ...
2
votes
1answer
83 views

Memory usage optimization for a large Association/SparseArray

Reformulating my question in simpler terms. Take two lists list1 = RandomInteger[{1, 100000000}, 25000000]; list2 = RandomReal[{1, 10}, 25000000]; Make an ...
2
votes
1answer
217 views

CUDALink ran out of available memory

I have a problem with automatic GPU memory management when using CUDA. The situation is the following: I create a simple CUDA function that operates on a 2D array (image), which does not take much ...
1
vote
0answers
58 views

Should I Remove the initial symbols of Module?

Often we use Module to construct complicate function. Consider that Module create new vars every time we run the function, should I remove the initial values at the end of Module-function?Or should I ...
0
votes
0answers
103 views

Out of Memory Error

I am currently constructing matrices, but I'm having problems with memory. When I evaluate some cells, the kernel quits due to lack of memory. I tried ...
0
votes
0answers
65 views

Handling really large Arrays (by possibly using lower precision) [duplicate]

I know there have been a few questions about efficient memory storage on mathematica.stackexchange, but I could not find a sufficient answer for my problem (except that it might not work at all). So ...
7
votes
2answers
239 views

How can I tell how much memory is taken by an expression?

ByteCount and LeafCount don't really work, as shown below. Mathematica 9.0 on a 32-bit Windows machine: ...
7
votes
2answers
112 views

Compress uses too much memory and sometimes crashes the Kernel

Using Compress is a good idea to save memory when working with large expressions. The problem is that Compressing large ...
3
votes
2answers
101 views

memory management in Mathematica

Unlike using Unprotect[In,Out]; Clear[In,Out]; Protect[In,Out]; to clean the occupied memory of the whole notebook, I want to clean the memory dynamically and ...
1
vote
0answers
44 views
15
votes
2answers
2k views

How do you deal with very large datasets in Mathematica?

This question is in two related parts. The first is about dataset size, the second data wrangling. Part 1: I've built a naive Bayesian classifier on Mathematica. It uses ...
1
vote
2answers
78 views

Table of ParametricPlots crashes Kernel

I can usually crash the Mathematica Kernel by using Table to generate a List of ...
0
votes
1answer
83 views

How do I manage my memory better when using the Map function?

So I'm attempting to write a Mathematica program that prints a range of natural numbers as the sum of four cubes. I wrote the following Mathematica code to try this, but am running into a memory ...
1
vote
0answers
76 views

Out of memory problem on constructing large list in Mathematica running on Linux [closed]

I have Mathematica running on Mac as well as Linux, my Mac as 16GB of RAM with core i5 processor (2-core) and Linux has 24GB of RAM on i7 (4-core). The problem is that the following command: ...
0
votes
0answers
125 views

Constructing and Reducing Tuples in a Memory Efficient Manner

This question is extension of a question I started here: Reduce the output from tuples by including symmetry? and will focus on higher order arrays and issues of optimization (similar to what I've ...
2
votes
0answers
63 views

Memory used by SparseArray objects

In dealing with a quantum mechanics problem, I am representing some many-body states in two different ways in Mathematica. For two parameters n and ...
4
votes
0answers
96 views

Memory leak when looping over FindInstance in version 10.0.0

When I run the following (simplified) code with FindInstance, my MemoryInUse is constantly increasing: ...
22
votes
3answers
833 views

DeleteDuplicatesBy is not performing as I'd hoped. Am I missing something?

Déjà vu: a new-in-v10 function should provide a better solution to an old problem, but my enthusiasm is curbed when I run timings. This time the function is ...
0
votes
0answers
115 views

What is the largest system of PDE you have solved with Mathematica?

I need to solve a system of partial differential equations. In the simplest case it consists of 2000 of PDE equations. In the most complex situation it consists up to 1 million of such equations. ...
2
votes
4answers
112 views

calculating subsets of a matrix (L=40X40)

I need to calculate all $k$-possible subsets$(k:1,L)$ of matrix with size $L$. I am using subset function as following, ...
1
vote
0answers
105 views

SystemException[MemoryAllocationFailure] with Memoized function [closed]

Version 10.1 Win7x64 Enterprise. This is more of a report than a question but others may have seen the same / have insights. I have some code (that I can't really post as it's too big) with a large ...
2
votes
1answer
145 views

mathematica use too much memory to Import large dat file

Err... When I try to Import a .dat file by dat = Import["D:\\TIEGCM\\solar_input\\omni_m_all_years.dat", "Table"]; or ...
3
votes
1answer
128 views

Memory leak in MemberQ

Bug introduced in 10.0.0 and fixed in 10.0.2 Consider the following code. ...
0
votes
1answer
72 views

What is stored in the kernel's memory

I disabled "Show Suggestions Bar after last output" in the preferences. I begin my code with $HistoryLength = 0. After running my code, I run the following. ...
21
votes
1answer
526 views

Excessive memory use by Mathematica kernel in 10.0.2 after executing simple commands

Update: The issue has been fixed in Mathematica 10.1! :) Warning: the code below may hang your system if you have the Suggestions Bar enabled in Mathematica < 10.1. When issuing the ...
5
votes
1answer
104 views

Comparing memory consumption of two pieces of code

I have implemented something I want to do in two different ways, because I'm not sure which of the two approaches give best speed/memory efficiency. Comparing the speed of the two approaches is ...
-1
votes
2answers
157 views

Want to compute the permutations of {1, 2, …, 11} with only 3 GB of memory [duplicate]

There is another way to calculate Permutations [{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}] without triggering an error, I have 3 GB of RAM with WIN 7 Edit: This ...
3
votes
2answers
99 views

Memory Problem with Modules and NDSolve

I'm using NDSolve inside a module, and I appear to have a memory leak. The relevant code is: ...
2
votes
0answers
48 views

Does Mathematica support out-of-core Associations?

I often find myself collecting data in an Association[] which outgrows my computer's main memory. Does Mathematica support any out-of-core equivalent? Ideally, I ...
2
votes
1answer
93 views

Simple-looking Minimize takes for ever

Consider the following: Minimize[{(ξ - x)^2 + (η - y)^2, a ξ + b η == c}, {ξ, η}, Reals] After asking Mathematica to do this yesterday afternoon and forgetting ...
0
votes
0answers
167 views

Is there a way to give Mathematica extra memory?

I am running Mathematica 10 (Student Edition) on Mac OS X 10.7.5 on an old Mac Pro. I have lots of extra RAM that the system isn't touching, and I would like to give some of it to Mathematica in the ...
8
votes
0answers
106 views

What is stored in FrontEnd memory?

I quit the kernel, and closed all notebooks, leaving only the startup window. Yet when I open a new notebook, MemoryInUse[$FrontEnd] reveals that the FrontEnd still ...
1
vote
0answers
84 views

Avoid copying memory for each subkernel

Here is a simplified example illustrating my issue; ...
10
votes
0answers
147 views

How can the Theil-Sen estimator be made to work on larger datasets?

The Theil-Sen estimator finds the slope and intercept of a line passing through a set of points by calculating the median slope and median intercept of the set of lines passing through all possible ...
5
votes
1answer
243 views

Using cursors with result sets to reduce memory usage

I have a database table with 900,000 rows and 80 columns per row which I'd like to work with in mathematica. My computations are isolated to each row, so I'd like to stream only a single row (or small ...
4
votes
1answer
91 views

Is there a way to constrain the memory used for a notebook (session)?

I know that we can constrain the memory usage of certain computation using MemoryConstrained. Is there a way to constrain the memory of each cell evaluation during ...
5
votes
1answer
234 views

Largest matrix size supported by Mathematica 10

I have need to store large square matrices in memory, some times sparse, some times not. I have used up to $2^{15}$ by $2^{15}$ as my upper bound so far, and Mathematica has handled it even if it took ...