Hot answers tagged

37

Scroll to the end for the latest edit. If you really want all the points in a large data set, I'd use a more basic display method: Let's say the matrix of data is m, then do this (assuming the entries m[[i, j]] are real numbers): Image[Rescale[m]] This will make one pixel for each data point without trying to do any interpolation as is the default in ...


35

Preamble It is hard to say what exactly is causing this without seeing the code, but, assuming that there are no memory leaks in the built-in functions you are using, I am only aware of a very few possible causes for memory leaks in Mathematica. Since almost anything is immutable, the leaks must be associated with some symbols for which definitions are ...


34

UPDATE I thought it would be neat to try and animate the thing, so I let the $a$ parameter run between $-\pi$ and $\pi$. I generated 600 images and put them together using ffmpeg. Check it out on youtube. It might not be in the spirit of Mathematica Stack Exchange, but allow me an objection - stuff that is slow in Mathematica should be kept out of it. To ...


34

The behavior described here is the same from 10.0.0 up to at least 10.3. Summary We can look at the code of DeleteDuplicatesBy and it turns out it uses GroupBy. The test cases proposed by Mr.Wizard are all handled by some part of the code of DeleteDuplicatesBy. Other parts of this code also seem to have some issues. Most of the members of the *By family of ...


33

As @RahulNarain says, forming the image point by point saves significant memory because the number of image pixels is typically much smaller than the hundreds of millions of iterations that compose it. Therefore, iterate the attractor equations, and for each point generated, find its location within the image matrix. Colour coding of the number of hits in ...


30

This is quite easy to achieve by direct manipulation of downvalues. Here's a simple example: ClearAll[removeDownValues]; SetAttributes[removeDownValues, HoldAllComplete]; removeDownValues[p : f_[___]] := DownValues[f] = DeleteCases[ DownValues[f, Sort -> False], HoldPattern[Verbatim[HoldPattern][p] :> _] ]; Now let's memoize some values: ...


29

Chunks of weak compositions Here is slightly modified version of algorithm used in Combinatorica`NextComposition converted to a LibraryFunction. Needs["CCompilerDriver`"] " #include \"WolframLibrary.h\" DLLEXPORT mint WolframLibrary_getVersion() { return WolframLibraryVersion; } DLLEXPORT int WolframLibrary_initialize(WolframLibraryData libData) { ...


27

You might be able to use JLink along with some undocumented behaviour of the Java class java.lang.management.ManagementFactory to get the information you seek: Needs["JLink`"] InstallJava[]; LoadJavaClass["java.lang.management.ManagementFactory"]; JavaBlock[ {#, java`lang`management`ManagementFactory`getOperatingSystemMXBean[]@#[]} & /@ { getName ,...


27

Is there a possibility to disable stack tracing, but keep messages? Internal`$MessageMenu = False reverts back to the old messages. Seems to do the trick and prevent the leak from my testing.


26

Since one may not always accurately predict when MemoryContrained is needed, I recommend setting up a watch-dog task. Belisarius described how to do this here in answer to my question. I will reproduce it below as answers that are merely links are discouraged. In Mathematica 8 you could start a memory watchdog, something along the lines of: ...


25

The technique is mentioned in one of Wolfram CDF virtual conference talks (See the course: Developing Real-World CDF Applications), as well as being used in a lot of CDF examples (for instance, the slideshow at the beginning of this example) but I will repeat it here, with some improvement. Recompressing images with better compression (Note: While writing ...


24

Analysis current as of Mathematica version 11.0.1 and 11.1.0. We can disable the Show Stack Trace item in the new message menu as follows: MessageMenu`$PruneStack; MessageMenu`Dump`$IncludeStack = False; The reference to MessageMenu`$PruneStack is there to ensure that the message menu packages have been autoloaded (otherwise our setting will be lost ...


23

I've always wondered about the scalability of MathLink (now officially "Wolfram Symbolic Transfer Protocol"). This is the protocol used by Mathematica to communicate between the front end and the kernel, and the basis of the Parallel` package. It has quite low bandwidth and high latency relative to, for example, MPI libraries. I also wonder how many MathLink ...


22

A very simple and straightforward test for square-freeness (and should be reasonably fast) is: squareFreeQ[str_] := StringFreeQ[str, x__ ~~ x__] Testing on your inputs: squareFreeQ["0101"] (* False*) squareFreeQ["0102012021"] (* True *) You can then possibly restrict this further to operate only on certain alphabets using Repeated and Alternatives. ...


22

Timing charts updated for version 10.0.2. The behavior remains unchanged. Attempting to analyze the performance of this function in the manner that Taliesin Beynon did for PositionIndex I shall use the same tools. The old method that will be compared in all cases below: myDeDupeBy[x_, f_] := GatherBy[x, f][[All, 1]] Speed A BenchmarkPlot of ...


21

Along the lines of Yamareth's answer, but probably an even better technique, is to put the following into $UserBaseDirectory/Kernel/init.m: Needs["JLink`"] SetOptions[InstallJava, JVMArguments->"-Xmx32g"] SetOptions[ReinstallJava, JVMArguments->"-Xmx32g"] ReinstallJava[] By setting the default options, you will get the desired heap size any time you (...


21

I revisited this problem - this time in pure Mathematica. The trick to any kind of performance is the Compile[] function, which in itself can be a bit moody - so you need to set global options to warn you when it refuses compilation and work around that. The performance I'm seeing is on the order of magnitude slower than that I get from C++, and two orders ...


21

I decided to take one of my large packages and Remove all symbols in the Private` context that have no definitions attached to them ... And is there any risk if only those symbols that are not used globally or that do not have definitions are carefully removed? In general, the complexity of the Mathematica language and the intricacies of the ...


21

Your function F is implemented really, really inefficiently. By quite simple means and in the proposed situation, it can be sped up by a factor of 20000. The key is to start with calculations in machine precision as early as possible and to store frequently used data in packed arrays. n = 100; mlist = Range[-1. + 2/n, 1. - 2/n, 2./n]; m2list = mlist^2; ...


20

Here is the method I outlined. I'll illustrate on a small example where we split matrix into top and bottom halves. In[794]:= SeedRandom[1111]; halfsize = 3; mat = RandomInteger[{-4, 4}, {2*halfsize, 10}] Out[796]= {{-3, -1, 3, -3, 3, 3, 3, 3, 4, 2}, {3, 3, -3, 0, 0, 1, -2, -4, 0, -1}, {-3, 4, 3, 0, -2, 4, 3, -2, -2, -2}, {2, 2, 4, 0, -4, 4, -1, -4, ...


19

There is something you can do. I just tried it with a JPEG image that I pasted into a notebook. The original image had 900kB and the notebook with only that image had 11.2 MB! Let's say you have an image in an output cell of your notebook, then select the cell bracket, go to Cell > Convert To... > Bitmap and the resulting notebook will be much ...


18

First off, you shouldn't be worried that Mathematica hasn't returned the memory to the system. The memory may as well be (and is being) held onto by the process that last had it, and if some other program needed this memory it would be handed back to the system without a problem. MaxMemoryUsed reports the maximum amount of memory used, but it is only ...


18

On my system (Windows 7 64-bit, 12GB, Mathematica v8) I only see a factor of 2 between the image file size and the memory used by the image data. This agrees with the observation that packed arrays of integers use 32 bits per element. To confirm this, a ConstantArray containing values of $2^{31}-1$ (the maximum signed 32-bit integer) is packed and has a ...


18

Preamble I will discuss here two methods for doing computations on very large data sets which don't fit into memory. The first method is based on sequential reading of chunks of data from a file. The second method is based on converting a data set to a file-backed list representation. The unifying idea for both methods is the use of iterators as a useful ...


18

Cause Under the hood System`SquaresR is still calling functions in the context NumberTheory`. Partial output of: Needs["GeneralUtilities`"] PrintDefinitions @ SquaresR SquaresR[2, NumberTheory`SquaresRDump`n _ Integer?Positive] := Block[{NumberTheory`SquaresRDump`res}, NumberTheory`SquaresRDump`res = NumberTheory`SquaresRDump`squaresR2[...


17

Preamble I will present a sort of a packaged and automated solution, which uses deques and metaprogramming to automate caching. This should work for most normal pattern-based functions. Deques I will use Daniel Lichtblau's implementation for a deque, taken from his great account on Data Structures and Efficient Algorithms in Mathematica. Here it is: ...


17

MyImagePartition In the meantime, before Mathematica 10 will come out, you can enjoy my on-foot solution MyImagePartition, which both saves memory and time using the PartitionMap function from the Developer context: MyImagePartition[im_, wh_, dwdh_List: {0, 0, 0}] := Module[{it = ImageType@im, cs = First@Options[im, ColorSpace], il = ...


17

$HistoryLength is just a global variable. It not clear the history. You can still access the history by ByteCount@Out[12] ByteCount@%10 It can be cleared by Unprotect[In, Out] Clear[In, Out] However, it would be better if you set $HistoryLength=0 before your resource-intensive code. P.S. It would be great to have $HistoryMemoryLimit or something like ...


17

Update: The issue has been fixed in Mathematica 10.1! :) Besides: Interesting enough, the suggestion bar's first option after evaluating the statements given above now is "solve for …", so: The assumption, that in 10.0.2 Mathematica's suggestion bar started out trying to actually solve for all variables, which, as I pointed out, is quite impossible ...


17

The memory leak in NIntegrate is a bug and has been fixed as of version 10.2.0. Earlier versions would lose around 720 bytes per evaluation for this example, which could not be recovered without restarting the kernel. ClearSystemCache[] should be used to make sure the memory is released. Using version 10.2: NI[z_?NumericQ, b0_?NumericQ] := NIntegrate[E^-...


Only top voted, non community-wiki answers of a minimum length are eligible