Tag Info

Hot answers tagged

22

If you really want all the points in a large data set, I'd use a more basic display method: Let's say the matrix of data is m, then do this (assuming the entries m[[i, j]] are real numbers): Image[Rescale[m]] This will make one pixel for each data point without trying to do any interpolation as is the default in density plots. Edit As suggested in the ...


22

A very simple and straightforward test for square-freeness (and should be reasonably fast) is: squareFreeQ[str_] := StringFreeQ[str, x__ ~~ x__] Testing on your inputs: squareFreeQ["0101"] (* False*) squareFreeQ["0102012021"] (* True *) You can then possibly restrict this further to operate only on certain alphabets using Repeated and Alternatives. ...


19

Preamble It is hard to say what exactly is causing this without seeing the code, but, assuming that there are no memory leaks in the built-in functions you are using, I am only aware of a very few possible causes for memory leaks in Mathematica. Since almost anything is immutable, the leaks must be associated with some symbols for which definitions are ...


19

You might be able to use JLink along with some undocumented behaviour of the Java class java.lang.management.ManagementFactory to get the information you seek: Needs["JLink`"] InstallJava[]; LoadJavaClass["java.lang.management.ManagementFactory"]; JavaBlock[ {#, java`lang`management`ManagementFactory`getOperatingSystemMXBean[]@#[]} & /@ { getName ...


17

Since one may not always accurately predict when MemoryContrained is needed, I recommend setting up a watch-dog task. Belisarius described how to do this here in answer to my question. I will reproduce it below as answers that are merely links are discouraged. In Mathematica 8 you could start a memory watchdog, something along the lines of: ...


17

On my system (Windows 7 64-bit, 12GB, Mathematica v8) I only see a factor of 2 between the image file size and the memory used by the image data. This agrees with the observation that packed arrays of integers use 32 bits per element. To confirm this, a ConstantArray containing values of $2^{31}-1$ (the maximum signed 32-bit integer) is packed and has a ...


17

This is quite easy to achieve by direct manipulation of downvalues. Here's a simple example: ClearAll[removeDownValues]; SetAttributes[removeDownValues, HoldAllComplete]; removeDownValues[p : f_[___]] := DownValues[f] = DeleteCases[ DownValues[f, Sort -> False], HoldPattern[Verbatim[HoldPattern][p] :> _] ]; Now let's memoize some ...


15

Here is the method I outlined. I'll illustrate on a small example where we split matrix into top and bottom halves. In[794]:= SeedRandom[1111]; halfsize = 3; mat = RandomInteger[{-4, 4}, {2*halfsize, 10}] Out[796]= {{-3, -1, 3, -3, 3, 3, 3, 3, 4, 2}, {3, 3, -3, 0, 0, 1, -2, -4, 0, -1}, {-3, 4, 3, 0, -2, 4, 3, -2, -2, -2}, {2, 2, 4, 0, -4, 4, -1, -4, ...


15

There is something you can do. I just tried it with a JPEG image that I pasted into a notebook. The original image had 900kB and the notebook with only that image had 11.2 MB! Let's say you have an image in an output cell of your notebook, then select the cell bracket, go to Cell > Convert To... > Bitmap and the resulting notebook will be much ...


14

The technique is mentioned in one of Wolfram CDF virtual conference talks (See the course: Developing Real-World CDF Applications), as well as being used in a lot of CDF examples (for instance, the slideshow at the beginning of this example) but I will repeat it here, with some improvement. Recompressing images with better compression (Note: While writing ...


14

MyImagePartition In the meantime, before Mathematica 10 will come out, you can enjoy my on-foot solution MyImagePartition, which both saves memory and time using the PartitionMap function from the Developer context: MyImagePartition[im_, wh_, dwdh_List: {0, 0, 0}] := Module[{it = ImageType@im, cs = First@Options[im, ColorSpace], il = ...


13

ListDensityPlot uses all the points in the dataset. You can set MaxPlotPoints option to control downsampling: ListDensityPlot[lst, MaxPlotPoints -> 1000] Memory consumption using MaxMemoryUsed: miu0 = MaxMemoryUsed[] lst = Table[Sin[x^2 + y^2], {x, 2^7}, {y, 2^13}]; usg0 = MaxMemoryUsed[] - miu0 miu1 = MaxMemoryUsed[] ListDensityPlot[lst, ...


13

You're looking to pack your array. It's done with Developer`ToPackedArray dataPacked = Developer`ToPackedArray[data]; ByteCount[dataPacked] 40168 There's more on this following the link. The links were taken from the packed array section of this answer.


13

In addition to Mr.Wizard's answer. In many cases it is very practical to stop the evaluation when the actual amount of free physical memory in your system becomes less than specified threshold. You can get the amount of free physical memory very efficiently via NETLink call to GlobalMemoryStatusEx function of kernel32.dll (which is available both under 32 ...


13

Preamble I will discuss here two methods for doing computations on very large data sets which don't fit into memory. The first method is based on sequential reading of chunks of data from a file. The second method is based on converting a data set to a file-backed list representation. The unifying idea for both methods is the use of iterators as a useful ...


13

This is not a direct answer to your question but it can help you. I see your question is about adaptive thresholding. I propose finding threshold values without exact partitioning. img = Import["http://homepages.inf.ed.ac.uk/rbf/HIPR2/images/son1.gif"] The simplest is GaussianFilter which is analog to $T = mean$ in your link. GF = GaussianFilter[img, ...


12

Have you tried setting $HistoryLength to zero: $HistoryLength=0; The memory is not freed, because the previous data is still available by Out[nn] or %nn. Note that if you do this, using %, %%, %n etc., which some people use in example code will not work. And when a large data set is shown the "Show More" and "Show Less" buttons do not work. You can ...


11

While we wait for an MMA implementation of BBP formula to generate the digits of Pi, we can use published results to identify repeated digits and their locations. Searching through the one billion digits of Pi in the file pi-billion.txt, in chunks of 10 million digits, with built-in function StringPosition: (patterns = Table[Table[i - 1, {9}], {i, 10}]; ...


11

You need to set $HistoryLength = 0 (or other small value) at the beginning of the session to prevent Out from remembering previous outputs. One non-obvious thing about Out is that if we do In[1]:= a=1; then Out[1] will still be set to 1 despite the semicolon at the end of the input! There's also the CleanSlate` package which has a ClearInOut function ...


11

Edit This one is much simpler than those I posted before . And very efficient Timing@StringFreeQ[benchmark, RegularExpression["(.+)\\1"]] Previous posts: Timings done on a VERY slow machine: Timing@Not@StringMatchQ[benchmark, RegularExpression[".*(.{1,1000})\\1(.*)"]] (* -> {0.735, True} *) Edit There is a problem if the repeated string has ...


11

If you need the ultimate speed, the following compiled code will be about 20 - 30 times faster than the elegant string-pattern based solution of @R.M. (but, of course, as many times longer and uglier): With[{part = Compile`GetElement}, squareFreeQLSC = Compile[{{ll, _Integer, 1}}, Module[{res = 0, ctr = 1, sctr = 1, len = 0, start = 0, i = 0}, ...


11

You can read lines from an InputStream strm (opened with OpenRead) in batches: lines=ReadList[strm, "String", 4000] You can vary the chunk size based on your application, 4000 is a number I found to work well for reading web server logs with lines that aren't crazy-long. You can also reposition for random access on startup. Version 9 improves the use of ...


11

Preamble I will present a sort of a packaged and automated solution, which uses deques and metaprogramming to automate caching. This should work for most normal pattern-based functions. Deques I will use Daniel Lichtblau's implementation for a deque, taken from his great account on Data Structures and Efficient Algorithms in Mathematica. Here it is: ...


11

Update This runs in about half a second: Total[1.0 ~Divide~ Flatten@Outer[Plus, Range[0, 8], Range[0, 80, 10], Range[0, 800, 100], Range[0, 8000, 1000], Range[0, 80000, 10000], Range[0, 800000, 100000], Range[0, 8000000, 1000000], Range[10000000, 80000000, 10000000]]] (* 1.07145 *) Original This is far from ...


10

I will only address a subquestion of yours, namely, "Does ParallelTable send a copy of the memory content to each parallel kernel?". The answer to that is a clear no: using ParallelTable does not mean that each kernel gets a copy of everything, but only of a subset of things (see the docs for DistributeDefinitions to see how the set of symbols to be ...


10

This should work ClearAll[MemoryHog]; MemoryHog[n_] := Module[{list}, list = {#, ByteCount[Out[#]]} & /@ Range[$Line]; list=SortBy[list, -Last[#] &]; list[[1 ;; n]] ]; Now lets test it (*Some sample input -- evaluate in three cells *) (*In[3]*) Range[1000]; (*In[4]*) Range[10000]; (*In[5]*) Range[100000]; (* get the hogs*) ...


10

This has [quadratic, he said] actually perhaps cubic complexity in a worst case (okay, now I'm just confused. More below).. Not the fastest of the lot, but it seems reasonable, or at least not entirely unreasonable. Requires some thought for me to see what I'm doing that keeps it relatively slow. squareFree[wrd_String] := squareFreeC[ToCharacterCode[wrd]] ...


10

Let me put my comment into an answer, because I think we might have misunderstood each other. You answered in the comment However, it would be great to do whole process (preparation of the matrix, solving the eigensystem, and further analysis) in Mathematica. That exactly was my idea. You only write some lines of C-Code which are compiled into a ...


9

You can put your calculation inside TimeConstrained. However in your case, probably the better idea is to restrict the used memory. That's done with MemoryConstrained. If you don't want to figure out the available memory yourself, see here for how to do it automatically. For example this terminates a calculation if the calculation needs more than 1 GB of ...


9

You can make use of either TimeConstrained or MemoryConstrained to terminate evaluation when it runs out of time or memory respectively. For example, if you have a function that has a reasonable memory footprint, but takes time to evaluate, you can abort evaluation after a certain amount of time (in seconds) has elapsed, as: ...



Only top voted, non community-wiki answers of a minimum length are eligible