# Tag Info

22

A very simple and straightforward test for square-freeness (and should be reasonably fast) is: squareFreeQ[str_] := StringFreeQ[str, x__ ~~ x__] Testing on your inputs: squareFreeQ["0101"] (* False*) squareFreeQ["0102012021"] (* True *) You can then possibly restrict this further to operate only on certain alphabets using Repeated and Alternatives. ...

18

You might be able to use JLink along with some undocumented behaviour of the Java class java.lang.management.ManagementFactory to get the information you seek: Needs["JLink"] InstallJava[]; LoadJavaClass["java.lang.management.ManagementFactory"]; JavaBlock[ {#, javalangmanagementManagementFactorygetOperatingSystemMXBean[]@#[]} & /@ { getName ...

18

If you really want all the points in a large data set, I'd use a more basic display method: Let's say the matrix of data is m, then do this (assuming the entries m[[i, j]] are real numbers): Image[Rescale[m]] This will make one pixel for each data point without trying to do any interpolation as is the default in density plots. Edit As suggested in the ...

17

Preamble It is hard to say what exactly is causing this without seeing the code, but, assuming that there are no memory leaks in the built-in functions you are using, I am only aware of a very few possible causes for memory leaks in Mathematica. Since almost anything is immutable, the leaks must be associated with some symbols for which definitions are ...

16

Since one may not always accurately predict when MemoryContrained is needed, I recommend setting up a watch-dog task. Belisarius described how to do this here in answer to my question. I will reproduce it below as answers that are merely links are discouraged. In Mathematica 8 you could start a memory watchdog, something along the lines of: ...

15

Here is the method I outlined. I'll illustrate on a small example where we split matrix into top and bottom halves. In[794]:= SeedRandom[1111]; halfsize = 3; mat = RandomInteger[{-4, 4}, {2*halfsize, 10}] Out[796]= {{-3, -1, 3, -3, 3, 3, 3, 3, 4, 2}, {3, 3, -3, 0, 0, 1, -2, -4, 0, -1}, {-3, 4, 3, 0, -2, 4, 3, -2, -2, -2}, {2, 2, 4, 0, -4, 4, -1, -4, ...

14

There is something you can do. I just tried it with a JPEG image that I pasted into a notebook. The original image had 900kB and the notebook with only that image had 11.2 MB! Let's say you have an image in an output cell of your notebook, then select the cell bracket, go to Cell > Convert To... > Bitmap and the resulting notebook will be much ...

14

This is quite easy to achieve by direct manipulation of downvalues. Here's a simple example: ClearAll[removeDownValues]; SetAttributes[removeDownValues, HoldAllComplete]; removeDownValues[p : f_[___]] := DownValues[f] = DeleteCases[ DownValues[f, Sort -> False], HoldPattern[Verbatim[HoldPattern][p] :> _] ]; Now let's memoize some ...

13

ListDensityPlot uses all the points in the dataset. You can set MaxPlotPoints option to control downsampling: ListDensityPlot[lst, MaxPlotPoints -> 1000] Memory consumption using MaxMemoryUsed: miu0 = MaxMemoryUsed[] lst = Table[Sin[x^2 + y^2], {x, 2^7}, {y, 2^13}]; usg0 = MaxMemoryUsed[] - miu0 miu1 = MaxMemoryUsed[] ListDensityPlot[lst, ...

13

You're looking to pack your array. It's done with DeveloperToPackedArray dataPacked = DeveloperToPackedArray[data]; ByteCount[dataPacked] 40168 There's more on this following the link. The links were taken from the packed array section of this answer.

13

The technique is mentioned in one of Wolfram CDF virtual conference talks (See the course: Developing Real-World CDF Applications), as well as being used in a lot of CDF examples (for instance, the slideshow at the beginning of this example) but I will repeat it here, with some improvement. Recompressing images with better compression (Note: While writing ...

13

On my system (Windows 7 64-bit, 12GB, Mathematica v8) I only see a factor of 2 between the image file size and the memory used by the image data. This agrees with the observation that packed arrays of integers use 32 bits per element. To confirm this, a ConstantArray containing values of $2^{31}-1$ (the maximum signed 32-bit integer) is packed and has a ...

13

Preamble I will discuss here two methods for doing computations on very large data sets which don't fit into memory. The first method is based on sequential reading of chunks of data from a file. The second method is based on converting a data set to a file-backed list representation. The unifying idea for both methods is the use of iterators as a useful ...

12

In addition to Mr.Wizard's answer. In many cases it is very practical to stop the evaluation when the actual amount of free physical memory in your system becomes less than specified threshold. You can get the amount of free physical memory very efficiently via NETLink call to GlobalMemoryStatusEx function of kernel32.dll. Here I define a getFreePhysMem[] ...

11

You need to set $HistoryLength = 0 (or other small value) at the beginning of the session to prevent Out from remembering previous outputs. One non-obvious thing about Out is that if we do In[1]:= a=1; then Out[1] will still be set to 1 despite the semicolon at the end of the input! There's also the CleanSlate package which has a ClearInOut function ... 11 Edit This one is much simpler than those I posted before . And very efficient Timing@StringFreeQ[benchmark, RegularExpression["(.+)\\1"]] Previous posts: Timings done on a VERY slow machine: Timing@Not@StringMatchQ[benchmark, RegularExpression[".*(.{1,1000})\\1(.*)"]] (* -> {0.735, True} *) Edit There is a problem if the repeated string has ... 11 If you need the ultimate speed, the following compiled code will be about 20 - 30 times faster than the elegant string-pattern based solution of @R.M. (but, of course, as many times longer and uglier): With[{part = Compile`GetElement}, squareFreeQLSC = Compile[{{ll, _Integer, 1}}, Module[{res = 0, ctr = 1, sctr = 1, len = 0, start = 0, i = 0}, ... 10 While we wait for an MMA implementation of BBP formula to generate the digits of Pi, we can use published results to identify repeated digits and their locations. Searching through the one billion digits of Pi in the file pi-billion.txt, in chunks of 10 million digits, with built-in function StringPosition: (patterns = Table[Table[i - 1, {9}], {i, 10}]; ... 10 As Leonid already commented, your code should not be especially slow. However one reason why your code may be slow is that you use Part to extract values, do some calculations and then insert the result. I would try using a wholesale approach and calculate all nodes at the same time by doing, e.g. (may need tweaking since I don't know the structure of your ... 10 I will only address a subquestion of yours, namely, "Does ParallelTable send a copy of the memory content to each parallel kernel?". The answer to that is a clear no: using ParallelTable does not mean that each kernel gets a copy of everything, but only of a subset of things (see the docs for DistributeDefinitions to see how the set of symbols to be ... 10 This should work ClearAll[MemoryHog]; MemoryHog[n_] := Module[{list}, list = {#, ByteCount[Out[#]]} & /@ Range[$Line]; list=SortBy[list, -Last[#] &]; list[[1 ;; n]] ]; Now lets test it (*Some sample input -- evaluate in three cells *) (*In[3]*) Range[1000]; (*In[4]*) Range[10000]; (*In[5]*) Range[100000]; (* get the hogs*) ...

10

This has [quadratic, he said] actually perhaps cubic complexity in a worst case (okay, now I'm just confused. More below).. Not the fastest of the lot, but it seems reasonable, or at least not entirely unreasonable. Requires some thought for me to see what I'm doing that keeps it relatively slow. squareFree[wrd_String] := squareFreeC[ToCharacterCode[wrd]] ...

9

Have you tried setting $HistoryLength to zero:$HistoryLength=0; The memory is not freed, because the previous data is still available by Out[nn] or %nn. Note that if you do this, using %, %%, %n etc., which some people use in example code will not work. And when a large data set is shown the "Show More" and "Show Less" buttons do not work. You can ...

9

You can read lines from an InputStream strm (opened with OpenRead) in batches: lines=ReadList[strm, "String", 4000] You can vary the chunk size based on your application, 4000 is a number I found to work well for reading web server logs with lines that aren't crazy-long. You can also reposition for random access on startup. Version 9 improves the use of ...

9

Preamble I will present a sort of a packaged and automated solution, which uses deques and metaprogramming to automate caching. This should work for most normal pattern-based functions. Deques I will use Daniel Lichtblau's implementation for a deque, taken from his great account on Data Structures and Efficient Algorithms in Mathematica. Here it is: ...

8

Others have given you some pointers about what's happening (and what not). I will focus on a simple self contained example which shows how you can set up a parallel computation which will not make copies of a large set of data but still make parts of that data accessible from the parallel kernels. Be aware that the approach I'm using here might have it's ...

8

You can put your calculation inside TimeConstrained. However in your case, probably the better idea is to restrict the used memory. That's done with MemoryConstrained. If you don't want to figure out the available memory yourself, see here for how to do it automatically. For example this terminates a calculation if the calculation needs more than 1 GB of ...

8

You can make use of either TimeConstrained or MemoryConstrained to terminate evaluation when it runs out of time or memory respectively. For example, if you have a function that has a reasonable memory footprint, but takes time to evaluate, you can abort evaluation after a certain amount of time (in seconds) has elapsed, as: ...

8

It's quite easy to show that Fold doesn't use the memory that would be required to store intermediate results. $HistoryLength = 0; big = Range[1*^7]; ByteCount[big] MaxMemoryUsed[] //N 40000124 5.49458*10^7 Fold[# + 1 &, big, Range@100]; MaxMemoryUsed[] //N 1.34909*10^8 FoldList by comparison (with a much shorter Range): FoldList[# + ... 8 Why do you have so many Out[] values assigned? This seems like a very poor practice if memory is a concern. I suggest setting something reasonable like$HistoryLength = 3 and finding another way to store your results. Nevertheless you can find the approximate memory used for each value like this: Array[{#, ByteCount@Out@#} &, \$Line - 1] This gives ...

Only top voted, non community-wiki answers of a minimum length are eligible