# Tag Info

29

Scroll to the end for the latest edit. If you really want all the points in a large data set, I'd use a more basic display method: Let's say the matrix of data is m, then do this (assuming the entries m[[i, j]] are real numbers): Image[Rescale[m]] This will make one pixel for each data point without trying to do any interpolation as is the default in ...

25

Summary We can look at the code of DeleteDuplicatesBy and it turns out it uses GroupBy. The test cases proposed by Mr.Wizard are all handled by some part of the code of DeleteDuplicatesBy. Other parts of this code also seem to have some issues. Most of the members of the *By family of functions seem to have side effects. How DeleteDuplicatesBy works It ...

23

Preamble It is hard to say what exactly is causing this without seeing the code, but, assuming that there are no memory leaks in the built-in functions you are using, I am only aware of a very few possible causes for memory leaks in Mathematica. Since almost anything is immutable, the leaks must be associated with some symbols for which definitions are ...

22

A very simple and straightforward test for square-freeness (and should be reasonably fast) is: squareFreeQ[str_] := StringFreeQ[str, x__ ~~ x__] Testing on your inputs: squareFreeQ["0101"] (* False*) squareFreeQ["0102012021"] (* True *) You can then possibly restrict this further to operate only on certain alphabets using Repeated and Alternatives. ...

22

As @RahulNarain says, forming the image point by point saves significant memory because the number of image pixels is typically much smaller than the hundreds of millions of iterations that compose it. Therefore, iterate the attractor equations, and for each point generated, find its location within the image matrix. Colour coding of the number of hits in ...

21

You might be able to use JLink along with some undocumented behaviour of the Java class java.lang.management.ManagementFactory to get the information you seek: Needs["JLink"] InstallJava[]; LoadJavaClass["java.lang.management.ManagementFactory"]; JavaBlock[ {#, javalangmanagementManagementFactorygetOperatingSystemMXBean[]@#[]} & /@ { getName ...

19

Since one may not always accurately predict when MemoryContrained is needed, I recommend setting up a watch-dog task. Belisarius described how to do this here in answer to my question. I will reproduce it below as answers that are merely links are discouraged. In Mathematica 8 you could start a memory watchdog, something along the lines of: ...

18

Timing charts updated for version 10.0.2. The behavior remains unchanged. Attempting to analyze the performance of this function in the manner that Taliesin Beynon did for PositionIndex I shall use the same tools. The old method that will be compared in all cases below: myDeDupeBy[x_, f_] := GatherBy[x, f][[All, 1]] Speed A BenchmarkPlot of ...

17

On my system (Windows 7 64-bit, 12GB, Mathematica v8) I only see a factor of 2 between the image file size and the memory used by the image data. This agrees with the observation that packed arrays of integers use 32 bits per element. To confirm this, a ConstantArray containing values of $2^{31}-1$ (the maximum signed 32-bit integer) is packed and has a ...

17

This is quite easy to achieve by direct manipulation of downvalues. Here's a simple example: ClearAll[removeDownValues]; SetAttributes[removeDownValues, HoldAllComplete]; removeDownValues[p : f_[___]] := DownValues[f] = DeleteCases[ DownValues[f, Sort -> False], HoldPattern[Verbatim[HoldPattern][p] :> _] ]; Now let's memoize some ...

16

First off, you shouldn't be worried that Mathematica hasn't returned the memory to the system. The memory may as well be (and is being) held onto by the process that last had it, and if some other program needed this memory it would be handed back to the system without a problem. MaxMemoryUsed reports the maximum amount of memory used, but it is only ...

16

UPDATE I thought it would be neat to try and animate the thing, so I let the $a$ parameter run between $-\pi$ and $\pi$. I generated 600 images and put them together using ffmpeg. Check it out on youtube. It might not be in the spirit of Mathematica Stack Exchange, but allow me an objection - stuff that is slow in Mathematica should be kept out of it. To ...

15

Here is the method I outlined. I'll illustrate on a small example where we split matrix into top and bottom halves. In[794]:= SeedRandom[1111]; halfsize = 3; mat = RandomInteger[{-4, 4}, {2*halfsize, 10}] Out[796]= {{-3, -1, 3, -3, 3, 3, 3, 3, 4, 2}, {3, 3, -3, 0, 0, 1, -2, -4, 0, -1}, {-3, 4, 3, 0, -2, 4, 3, -2, -2, -2}, {2, 2, 4, 0, -4, 4, -1, -4, ...

15

There is something you can do. I just tried it with a JPEG image that I pasted into a notebook. The original image had 900kB and the notebook with only that image had 11.2 MB! Let's say you have an image in an output cell of your notebook, then select the cell bracket, go to Cell > Convert To... > Bitmap and the resulting notebook will be much ...

14

In addition to Mr.Wizard's answer. In many cases it is very practical to stop the evaluation when the actual amount of free physical memory in your system becomes less than specified threshold. You can get the amount of free physical memory very efficiently via NETLink call to GlobalMemoryStatusEx function of kernel32.dll (which is available both under 32 ...

14

The technique is mentioned in one of Wolfram CDF virtual conference talks (See the course: Developing Real-World CDF Applications), as well as being used in a lot of CDF examples (for instance, the slideshow at the beginning of this example) but I will repeat it here, with some improvement. Recompressing images with better compression (Note: While writing ...

14

You're looking to pack your array. It's done with DeveloperToPackedArray dataPacked = DeveloperToPackedArray[data]; ByteCount[dataPacked] 40168 There's more on this following the link. The links were taken from the packed array section of this answer.

14

Along the lines of Yamareth's answer, but probably an even better technique, is to put the following into $UserBaseDirectory/Kernel/init.m: Needs["JLink"] SetOptions[InstallJava, JVMArguments->"-Xmx32g"] SetOptions[ReinstallJava, JVMArguments->"-Xmx32g"] ReinstallJava[] By setting the default options, you will get the desired heap size any time you ... 14 MyImagePartition In the meantime, before Mathematica 10 will come out, you can enjoy my on-foot solution MyImagePartition, which both saves memory and time using the PartitionMap function from the Developer context: MyImagePartition[im_, wh_, dwdh_List: {0, 0, 0}] := Module[{it = ImageType@im, cs = First@Options[im, ColorSpace], il = ... 13 ListDensityPlot uses all the points in the dataset. You can set MaxPlotPoints option to control downsampling: ListDensityPlot[lst, MaxPlotPoints -> 1000] Memory consumption using MaxMemoryUsed: miu0 = MaxMemoryUsed[] lst = Table[Sin[x^2 + y^2], {x, 2^7}, {y, 2^13}]; usg0 = MaxMemoryUsed[] - miu0 miu1 = MaxMemoryUsed[] ListDensityPlot[lst, ... 13 Preamble I will discuss here two methods for doing computations on very large data sets which don't fit into memory. The first method is based on sequential reading of chunks of data from a file. The second method is based on converting a data set to a file-backed list representation. The unifying idea for both methods is the use of iterators as a useful ... 13 This is not a direct answer to your question but it can help you. I see your question is about adaptive thresholding. I propose finding threshold values without exact partitioning. img = Import["http://homepages.inf.ed.ac.uk/rbf/HIPR2/images/son1.gif"] The simplest is GaussianFilter which is analog to$T = mean$in your link. GF = GaussianFilter[img, ... 13 Adding up all your kind comments, we can conclude: An enabled Suggestion Bar (standard in Mathematica 10) triggers one WolframKernel process to consume all system memory (possibly up to approximately 4.37 TB, if available). This is a bug in Mathematica 10 (up to and including 10.0.2.0) on multiple operating system platforms. Wolfram has confirmed the ... 12 You need to set$HistoryLength = 0 (or other small value) at the beginning of the session to prevent Out from remembering previous outputs. One non-obvious thing about Out is that if we do In[1]:= a=1; then Out[1] will still be set to 1 despite the semicolon at the end of the input! There's also the CleanSlate package which has a ClearInOut function ...

12

Have you tried setting $HistoryLength to zero:$HistoryLength=0; The memory is not freed, because the previous data is still available by Out[nn] or %nn. Note that if you do this, using %, %%, %n etc., which some people use in example code will not work. And when a large data set is shown the "Show More" and "Show Less" buttons do not work. You can ...

12

$HistoryLength is just a global variable. It not clear the history. You can still access the history by ByteCount@Out[12] ByteCount@%10 It can be cleared by Unprotect[In, Out] Clear[In, Out] However, it would be better if you set$HistoryLength=0 before your resource-intensive code. P.S. It would be great to have \$HistoryMemoryLimit or something like ...

11

Edit This one is much simpler than those I posted before . And very efficient Timing@StringFreeQ[benchmark, RegularExpression["(.+)\\1"]] Previous posts: Timings done on a VERY slow machine: Timing@Not@StringMatchQ[benchmark, RegularExpression[".*(.{1,1000})\\1(.*)"]] (* -> {0.735, True} *) Edit There is a problem if the repeated string has ...

11

If you need the ultimate speed, the following compiled code will be about 20 - 30 times faster than the elegant string-pattern based solution of @R.M. (but, of course, as many times longer and uglier): With[{part = CompileGetElement}, squareFreeQLSC = Compile[{{ll, _Integer, 1}}, Module[{res = 0, ctr = 1, sctr = 1, len = 0, start = 0, i = 0}, ...

11

While we wait for an MMA implementation of BBP formula to generate the digits of Pi, we can use published results to identify repeated digits and their locations. Searching through the one billion digits of Pi in the file pi-billion.txt, in chunks of 10 million digits, with built-in function StringPosition: (patterns = Table[Table[i - 1, {9}], {i, 10}]; ...

11

You can read lines from an InputStream strm (opened with OpenRead) in batches: lines=ReadList[strm, "String", 4000] You can vary the chunk size based on your application, 4000 is a number I found to work well for reading web server logs with lines that aren't crazy-long. You can also reposition for random access on startup. Version 9 improves the use of ...

Only top voted, non community-wiki answers of a minimum length are eligible