Are there advantages to using additional arguments of common functions rather than alternative ways of calculating with lists?

Question

(Apologies for the long question title.)

One of the interesting, if sometimes confusing, things about Mathematica is that there is always more than one way to do things. Even intermediate users can be confronted with the sudden realisation that there was an alternative syntax to do the same thing. There seem to be a number of less-well-known parts of the language hiding in plain sight as additional arguments to some very well known functions. Consider the following examples:

1. Total: Total[somematrix, {2}] is equivalent to
   Total /@ somematrix
2. Partition: Divide @@@ Partition[somevector, 2, 1] is equivalent to
   Most[somevector]/Rest[somevector]
3. Join: Join[somematrixwith10rows, Array[f, {10, 1}], 2] is equivalent to some complicated thing involving Append or Transpose to add a column at the end.

I am sure there are others.

The common thread between all three of the first example in each pair above is an additional argument that is often not well known, at least by me.

Are there reasons for preferring one style over the other? It has been suggested to me that at least in the first example, the additional-argument version doesn't unpack and therefore saves memory over the case using Map. But are there other reasons, for example the ability to Compile, or other efficiency improvements?

If it all down to personal taste and knowledge, I worry that it makes it hard to understand and learn from other people's code. And if there are best practices to follow, it would be good to make them more widely known.

Answer 1

I can think of several main advantages of additional arguments:

• Efficiency
• More concise and readable code
• Better abstraction level
• Less chances for bugs.

In brief, I think that the first reason is bad most of the time, the second and third are valid, and the last may or may not be valid. Let us now consider these arguments.

I start with efficiency. My opinion here is that, on one hand, this is one of the major factors in real-life Mathematica programming (or at least, in my experience), while on the other hand, it should not be. I really blame the execution model of Mathematica here (yes, I know many reasons why with Mathematica it can not currently be the other way, but, from the pragmatic point of view of the user of it as a general-purpose programming language - as opposed to a computer algebra system or rewrite engine - I could not care less), because the performance differences between implemented in C built-in functions and top-level user-defined functions are often huge, and this makes performance non-uniform and hard to understand. We all spend lots of time on micro-benchmarking, while that time could have been spent much more productively on some really important things. So, while in practice the performance factor often dominates the decision to use these extra arguments (or even introduce them in the function's syntax), I think this is conceptually wrong one and should have a status of a widely used work-around to compensate for the current language limitations regarding performance. A real way out would IMO be to extend Compile so that much wider subset of the language could become compiled rather than interpreted, which is of course a much harder task.

Next point is concise and readable code. This one is a very valid one IMO. However, using extra arguments does not always lead to that, because often what they do is rather obscure. So, I think that some balance is needed here. It also depends on how well these extra arguments are designed, in the sense that it should be relatively easy to understand what they do in some code, perhaps by consulting the docs. In any case, there are IMO many use cases where the use of extra arguments may be justified from this point of view. Good examples here are IMO SortBy and SplitBy, particularly with their extended (multi-level) functionality (they also may improve performance, but I don't view this as their major advantage).

What I mean by better abstraction level is that extra arguments may allow to group a lot of related functionality into one "super-function" (e.g. Partition), which means that the code using it does recognize in the particular use of it an instance of a more general operation, conceptually. This may be useful in the same way as abstract classes are useful in OOP. This however requires a really careful and very well thought out design for such functions. There are quite a few Mathematica functions which, I think, satisfy this criteria.

Less chances for bugs: since built-ins are used (assuming that built-ins are having on the average less bugs than the code you write, which is probably true for most users, because built-ins were written by experts, and had more chances to be tested, being exposed to many users and undergone the QA precess). This may not be as clear-cut, however, in cases when these extra arguments are hard to understand, since using them may induce more bugs because of that.

So, my conclusions would be these: if the code using those arguments is more readable, natural and concise (this is subjective, of course), they are probably worth using, in that particular piece of code. I have seen many instances where this is indeed the case. But, if all they give is a speed increase, especially at the expense of code clarity, they are probably not worth using (or at least, it is not worth to spend excessive amounts of time searching for them and reorganizing code), except when this piece of code is speed-critical.

A very large class of use cases when they are used to only boost the performance I view as a necessary evil, but, while I think that the expert-level performance-tuning skills must currently include the knowledge of these arguments / use cases, I also think that this is unfortunate and leads to massive learning / memorizing effort on the side of a practitioner, which could be better spent elsewhere, were the language performance model more uniform. I actually don't think that this problem is characteristic to Mathematica only, most high-level languages probably have similar issues. What makes Mathematica stand out IMO is that, being extremely high-level, it has a very large range of the possible performance differences between different solutions, which may easily span several orders of magnitude.