# Tag Info

34

UPDATE I thought it would be neat to try and animate the thing, so I let the $a$ parameter run between $-\pi$ and $\pi$. I generated 600 images and put them together using ffmpeg. Check it out on youtube. It might not be in the spirit of Mathematica Stack Exchange, but allow me an objection - stuff that is slow in Mathematica should be kept out of it. To ...

34

The behavior described here is the same from 10.0.0 up to at least 10.3. Summary We can look at the code of DeleteDuplicatesBy and it turns out it uses GroupBy. The test cases proposed by Mr.Wizard are all handled by some part of the code of DeleteDuplicatesBy. Other parts of this code also seem to have some issues. Most of the members of the *By family of ...

33

As @RahulNarain says, forming the image point by point saves significant memory because the number of image pixels is typically much smaller than the hundreds of millions of iterations that compose it. Therefore, iterate the attractor equations, and for each point generated, find its location within the image matrix. Colour coding of the number of hits in ...

30

This is quite easy to achieve by direct manipulation of downvalues. Here's a simple example: ClearAll[removeDownValues]; SetAttributes[removeDownValues, HoldAllComplete]; removeDownValues[p : f_[___]] := DownValues[f] = DeleteCases[ DownValues[f, Sort -> False], HoldPattern[Verbatim[HoldPattern][p] :> _] ]; Now let's memoize some values: ...

30

Chunks of weak compositions Here is slightly modified version of algorithm used in CombinatoricaNextComposition converted to a LibraryFunction. Needs["CCompilerDriver"] " #include \"WolframLibrary.h\" DLLEXPORT mint WolframLibrary_getVersion() { return WolframLibraryVersion; } DLLEXPORT int WolframLibrary_initialize(WolframLibraryData libData) { ...

28

Is there a possibility to disable stack tracing, but keep messages? Internal$MessageMenu = False reverts back to the old messages. Seems to do the trick and prevent the leak from my testing. 24 Analysis current as of Mathematica version 11.0.1 and 11.1.0. We can disable the Show Stack Trace item in the new message menu as follows: MessageMenu$PruneStack; MessageMenuDump$IncludeStack = False; The reference to MessageMenu$PruneStack is there to ensure that the message menu packages have been autoloaded (otherwise our setting will be lost ...

23

I've always wondered about the scalability of MathLink (now officially "Wolfram Symbolic Transfer Protocol"). This is the protocol used by Mathematica to communicate between the front end and the kernel, and the basis of the Parallel package. It has quite low bandwidth and high latency relative to, for example, MPI libraries. I also wonder how many MathLink ...

22

I revisited this problem - this time in pure Mathematica. The trick to any kind of performance is the Compile[] function, which in itself can be a bit moody - so you need to set global options to warn you when it refuses compilation and work around that. The performance I'm seeing is on the order of magnitude slower than that I get from C++, and two orders ...

22

Timing charts updated for version 10.0.2. The behavior remains unchanged. Attempting to analyze the performance of this function in the manner that Taliesin Beynon did for PositionIndex I shall use the same tools. The old method that will be compared in all cases below: myDeDupeBy[x_, f_] := GatherBy[x, f][[All, 1]] Speed A BenchmarkPlot of ...

21

Along the lines of Yamareth's answer, but probably an even better technique, is to put the following into $UserBaseDirectory/Kernel/init.m: Needs["JLink"] SetOptions[InstallJava, JVMArguments->"-Xmx32g"] SetOptions[ReinstallJava, JVMArguments->"-Xmx32g"] ReinstallJava[] By setting the default options, you will get the desired heap size any time you (... 21 I decided to take one of my large packages and Remove all symbols in the Private context that have no definitions attached to them ... And is there any risk if only those symbols that are not used globally or that do not have definitions are carefully removed? In general, the complexity of the Mathematica language and the intricacies of the evaluation ... 21 Your function F is implemented really, really inefficiently. By quite simple means and in the proposed situation, it can be sped up by a factor of 20000. The key is to start with calculations in machine precision as early as possible and to store frequently used data in packed arrays. n = 100; mlist = Range[-1. + 2/n, 1. - 2/n, 2./n]; m2list = mlist^2; ... 18 On my system (Windows 7 64-bit, 12GB, Mathematica v8) I only see a factor of 2 between the image file size and the memory used by the image data. This agrees with the observation that packed arrays of integers use 32 bits per element. To confirm this, a ConstantArray containing values of$2^{31}-1$(the maximum signed 32-bit integer) is packed and has a ... 18 Preamble I will discuss here two methods for doing computations on very large data sets which don't fit into memory. The first method is based on sequential reading of chunks of data from a file. The second method is based on converting a data set to a file-backed list representation. The unifying idea for both methods is the use of iterators as a useful ... 18 Cause Under the hood SystemSquaresR is still calling functions in the context NumberTheory. Partial output of: Needs["GeneralUtilities"] PrintDefinitions @ SquaresR SquaresR[2, NumberTheorySquaresRDumpn _ Integer?Positive] := Block[{NumberTheorySquaresRDumpres}, NumberTheorySquaresRDumpres = NumberTheorySquaresRDump`squaresR2[... 17 Preamble I will present a sort of a packaged and automated solution, which uses deques and metaprogramming to automate caching. This should work for most normal pattern-based functions. Deques I will use Daniel Lichtblau's implementation for a deque, taken from his great account on Data Structures and Efficient Algorithms in Mathematica. Here it is: ... 17 MyImagePartition In the meantime, before Mathematica 10 will come out, you can enjoy my on-foot solution MyImagePartition, which both saves memory and time using the PartitionMap function from the Developer context: MyImagePartition[im_, wh_, dwdh_List: {0, 0, 0}] := Module[{it = ImageType@im, cs = First@Options[im, ColorSpace], il = ... 17 ClearAll clears all definitions associated with the symbol. However, the symbol remains in the symbol table, so all references to that symbol from other symbols (their definitions) remain fully valid. The symbol can then acquire new rules or other global properties associated with it. Remove removes the symbol from the symbol table. More precisely, it ... 17$HistoryLength is just a global variable. It not clear the history. You can still access the history by ByteCount@Out[12] ByteCount@%10 It can be cleared by Unprotect[In, Out] Clear[In, Out] However, it would be better if you set $HistoryLength=0 before your resource-intensive code. P.S. It would be great to have$HistoryMemoryLimit or something like ...

17

Update: The issue has been fixed in Mathematica 10.1! :) Besides: Interesting enough, the suggestion bar's first option after evaluating the statements given above now is "solve for …", so: The assumption, that in 10.0.2 Mathematica's suggestion bar started out trying to actually solve for all variables, which, as I pointed out, is quite impossible ...

17

The memory leak in NIntegrate is a bug and has been fixed as of version 10.2.0. Earlier versions would lose around 720 bytes per evaluation for this example, which could not be recovered without restarting the kernel. ClearSystemCache[] should be used to make sure the memory is released. Using version 10.2: NI[z_?NumericQ, b0_?NumericQ] := NIntegrate[E^-...

17

I suggest not to use Graph to render, as it is slow. Instead, compute the vertex coordinates, then render manually. The rough steps are as follows: Create an "index graph", i.e. a graph where the vertex names coincide with the vertex indices: n = 10000; g = IndexGraph@Graph[ Flatten[(Rule @@@ Partition[#, 2, 1]) & /@ CollatzSequence@Range[n]...

15

For individual cases I believe the most straight forward solution is simply using Unset: For instance: f[x_] := f[x] = x f[1]; f[2]; f[5]; DownValues[f] f[5] =. f[3]; DownValues[f] (* {HoldPattern[f[1]] :> 1, HoldPattern[f[2]] :> 2, HoldPattern[f[5]] :> 5, HoldPattern[f[x_]] :> (f[x] = x)} *) (* {HoldPattern[f[1]] :> 1, ...

15

I am sure you can easily install also Linux on it and then you could contact Vladyslav Shtabovenko, the current maintainer of FeynCalc (https://github.com/vsht) and ask him about hard problems in High Energy Physics he would like to benchmark on such a King-Kong machine. Either him or somebody else could also provide you with more complicated examples of ...

14

You can use ByteCount to look at the memory usage. ByteCount is not precise (doesn't take into account sharing), but it will make it easier to understand what is going on. Let's do this experiment: In[1]:= list = Subsets[Range[100], {3}]; In[2]:= ByteCount[list] Out[2]= 19404080 In[3]:= ByteCount[Transpose[list]] Out[3]= 11642704 Note that Transpose[...

14

You can read lines from an InputStream strm (opened with OpenRead) in batches: lines=ReadList[strm, "String", 4000] You can vary the chunk size based on your application, 4000 is a number I found to work well for reading web server logs with lines that aren't crazy-long. You can also reposition for random access on startup. Version 9 improves the use of ...

14

Your explanations are very detailed and I have to admit it's too detailed for me to go through it, trying to understand without a minimal working example. Therefore, view this as some ideas for your first, short-version question. I will present two different approaches, where the first one takes very long, but consumes almost no memory and the second one is ...

14

This is not a direct answer to your question but it can help you. I see your question is about adaptive thresholding. I propose finding threshold values without exact partitioning. img = Import["http://homepages.inf.ed.ac.uk/rbf/HIPR2/images/son1.gif"] The simplest is GaussianFilter which is analog to $T = mean$ in your link. GF = GaussianFilter[img, 30] ...

14

This function will be rewritten in C for 10.0.2 and should come down to average-case complexity of $O(n)$ from its current $O(n \log(n))$. Note that the version most users will be bothered to write (and the way we advertized this before in the docpage for DeleteDuplicates) is $O(n^2)$, so most users are probably already winning. In the meantime, my advice ...

Only top voted, non community-wiki answers of a minimum length are eligible