ByteCount and LeafCount don't really work, as shown below.

Mathematica 9.0 on a 32-bit Windows machine:

f[n_] := Block[{k = f[n - 1]}, {k, k}]
f[0] = x

LeafCount[f[k]] for particular small integers k returns the number 2^(k+1) - 1 e.g.
for k = 3, it returns 15. But it returns the same answer for k = 30, 31, .... It returns 2147483647, which, probably not coincidentally, is 2^31 - 1.

ByteCount has a similar problem in running out of bits in its counter.

Anyway, ByteCount also lies in a different way… because it doesn't take into account the sharing that (say) f[70] has. Without the sharing implicit in the formula above, I could not compute that — it has $2.36 \times 10^{21}$ leaves. They are just not different leaves.

Presumably, techniques that produce expressions with smaller memory footprints should be preferable.

  • 1
    $\begingroup$ What do you get from evaluating Developer`$MaxMachineInteger? I suspect this is a 32-bit thing. $\endgroup$ – chuy May 8 '14 at 16:35

If ByteCount is to be considered lying, it is at least documented lying:

ByteCount does not take account of any sharing of subexpressions. The results it gives assume that every part of the expression is stored separately. ByteCount will therefore often give an overestimate of the amount of memory currently needed to store a particular expression. When you manipulate the expression, however, subexpressions will often stop being shared, and the amount of memory needed will be close to the value returned by ByteCount. >>

The doc link there at the end points to this example at the same ref page:

ByteCount does not account for sharing in the actual storage of expressions:

In[1]:= x = 1000!;

In[2]:= ByteCount[x]

Out[2]= 1096

In[3]:= ByteCount[{x, x, x}]

Out[3]= 3320

Counting "actual" bytes

Here's how I (try to) check for actual byte counts:

realByteCount[expr_] := 
 ByteCount@expr - Share@MapAll[Identity, expr, Heads -> True]


In[8]:= realByteCount@f[20]

Out[8]= 1120

Keep in mind that the MapAll expression unshares the whole expression. I.e. if ByteCount was reporting 2 GiB for your expression, you'd better have more than 2 GiB free (and be running a 64-bit process).

ByteCount/LeafCount counter maximum

Re the behavior with the internal leaf or byte counter hitting a maximum, I'm not sure how I'd work around it. For the sake of including a note about it in the documentation, it might be worth a note to WRI Support (unless it's already in the docs and I just haven't seen it yet).


If it's really crucial to overcome this obstacle, you could try the following:

nomaxCount[e_, f : ByteCount | LeafCount] :=
 nomaxCount[e, f] =
  With[{bc = f@e},
   If[bc === Developer`$MaxMachineInteger,
    With[{bcs = nomaxCount[#, f] & /@ e}, 
     Total@bcs + f@bcs - Total[f /@ bcs]

Here's an example of using it:

In[62]:= nomaxCount[f[27], ByteCount]

Out[62]= 5368709080

In[63]:= nomaxCount[f[31], LeafCount] // AbsoluteTiming

Out[63]= {708.838881, 4294967295}

It's slower than the originals at counts this high, but it does do the job. (Frankly, by my testing I'm not totally sure the memoization is helping, either. Perhaps the kernel is already taking care of it.)

Even so, I believe that if ByteCount/LeafCount were adjusted to use an arbitrary-precision integer count, they'd be in all cases only slightly slower than they are now and would be much faster in above-$MaxMachineInteger counts than the workaround I've offered up above.

And finally...

I went ahead and filed a suggestion with WRI Support for them to maybe consider adjusting ByteCount/LeafCount to use an arbitrary-precision count.

It seems like the kind of thing that might get back-burnered as not very critical to most users, but hey, it's kind of a broken window, if only a small one, and it should be at least brought to their attention.

I also suggested a built-in function like realByteCount above, except that ideally it wouldn't unshare the whole expression to determine its with-sharing memory footprint.

  • $\begingroup$ I tried altByteCounter[f[31]], and after a considerable pause, the kernel crashed. $\endgroup$ – Richard Fateman May 8 '14 at 17:59
  • $\begingroup$ @RichardFateman, I'm pretty sure that's because f[31] has an unshared ByteCount of 2 GiB and the kernel process couldn't acquire that much memory on your 32-bit system. See msdn.microsoft.com/en-us/library/windows/desktop/…. The unshared ByteCount is realized by the MapAll expression, which touches all parts of the expression and unshares them. There's a bit more ref material here: reference.wolfram.com/mathematica/tutorial/… $\endgroup$ – William May 8 '14 at 23:28
  • $\begingroup$ @RichardFateman, actually I misspoke, f[31] must have a ByteCount much larger than that, since f[26] is where I see the maxed-out ByteCount counter on my own 32-bit Linux system. I've played with this a little more, and I believe there's a way to bypass both the massive unsharing caused by my use of MapAll as well as the ByteCount counter limit. For LeafCount it might work similarly. I'll update this answer when I have a chance. $\endgroup$ – William May 8 '14 at 23:51
  • $\begingroup$ @RichardFateman, I found a way to overcome the counter limit, if you're still interested in a way around that. Unfortunately, I couldn't come up with a way to do the "actual"-ByteCount in any reasonably correct way without unsharing the expression and thus blowing things up when the expression is too large to fit in memory. For smaller expressions, though, I've found it pretty useful. $\endgroup$ – William May 9 '14 at 2:36
  • $\begingroup$ There are counters in some hardware used for profiling etc. Since you can easily exceed 2^32 machine cycles, the counters tend to be several words long on a 32-bit machine. You don't need arb. precision. You could use 32-bits + overflow detection. Still, the counting is wrong and could be done right for unique bytes with some tricks. $\endgroup$ – Richard Fateman May 9 '14 at 15:21

Two methods come to mind.

The first is to find the size of (a string expressing) the representation of the expression as it exists internally to Mathematica:

 StringLength@ExportString[f[k], "MX"], 
 {k, 0, 20}
] // ListLinePlot

Plot showing linear dependence on k of the size of a dump string

The main disadvantage of this is that the result is only linearly dependent on (not equal to) the true answer, because the string being measured actually contains a serialized version of the internal representation. Also, it can be very slow for huge expressions (such as one encounters with large values of k, where "large" in this case means bigger than about 25), probably due to some overhead of the serialization process such as having to walk the entire expression tree.

The second method is undocumented, and was implemented for Oliver Ruebenkoenig (who provided the first description of it, here):

 {k, 0, 20}
] // ListLinePlot

Plot showing linear dependence on k of the result of MaxMemoryUsed

This should be more representative of the actual in-memory size of an expression, and returns its result quickly (for example, it can tell us that f[1000] takes up about 305KB on a 64-bit system and 211KB on a 32-bit one, which seems quite sensible). But, as it's undocumented, there is of course more reason than usual to doubt whether it will always produce the right answer. On the other hand, we do know that this number comes straight from the memory allocator, which (one hopes) should know what it's talking about.

  • $\begingroup$ The undocumented MaxMemoryUsed[f[k]] does the job. Thanks. $\endgroup$ – Richard Fateman May 11 '14 at 3:16
  • 2
    $\begingroup$ @RichardFateman You're welcome. Please don't forget to vote on questions and answers (arrows in the top left corner of each post), and, if you want to declare that the question is answered, pick an accepted answer (check mark below the arrows). Voting is what drives the entire StackExchange economy, so it's important! (By the way, it also isn't necessarily a good idea to tag questions with the names of the functions under discussion. Questions are searchable by their full text already, and tags are intended as a higher level of abstraction.) $\endgroup$ – Oleksandr R. May 11 '14 at 3:29
  • $\begingroup$ @RichardFateman just came across this one, you might consider accepting Oleksandr´s answer to (amongst other things) mark this question as properly answered. $\endgroup$ – Yves Klett Jun 10 '14 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.