Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

While reading Leonid's grand answers to General strategies to write big code in Mathematica? I came across something that goes against my own practices. I do not disagree with the principle but the degree to which it is taken feels both alien and counterproductive to me. Quite possibly Leonid is right, he usually is, but I wish to indulge in a counterargument even if it ultimately only proves his point.

He gives as his example of granular coding this:

ClearAll[returnedQ,randomSteps,runExperiment,allReturns,stats];

returnedQ[v_,steps_]:=MemberQ[Accumulate[v[[steps]]],{0,0}];

randomSteps[probs_,q_]:=RandomChoice[probs->Range[Length[probs]],q];

runExperiment[v_,probs_,q_]:= returnedQ[v,randomSteps[probs,q]];

allReturns[n_,q_,v_,probs_]:= Total @ Boole @ Table[runExperiment[v,probs,q],{n}]

stats[z_,n_,q_,v_,probs_]:=Table[allReturns[n,q,v,probs],{z}];

I have expressly left out the explanatory comments. Answering questions on Stack Exchange has taught me that code often doesn't do what descriptions claim it does, and it is better to read and understand the code itself for a true understanding.

I find the level of granularity illustrated above distracting rather than illuminating.

  • There is quite a lot of abstract fluff in the form of function names to tell me what code does rather that just showing me what it does in simple, reabable steps.

  • Each subfunction has multiple parameters and the relationship between these functions is not clear at a glance. The evaluation order ultimately proves simple but the code itself feels convoluted.

  • To follow this code I have to read it backwards, working inside out, and I have to keep track of multiple arguments at each step. Leonid wisely keeps the parameters consistent throughout but this cannot be assumed at first read, therefore additional mental effort must be expended.

Conversely in my own terse paradigm I would write the function as follows:

stats2[z_, n_, q_, v_, probs_] :=
  With[{freq = probs -> Range @ Length @ probs},
    (
      v[[ freq ~RandomChoice~ q ]]
        // Accumulate
        // MemberQ[{0, 0}]
        // Boole
     ) ~Sum~ {n} ~Table~ {z}
  ];

I find this greatly superior for personal ease of reading and comprehension.

I know that my style is unconventional and at times controversial; some no doubt flinch at my use of ~infix~ operators. Nevertheless I stand by my assertion that once this becomes familiar the code is very easy to read.

  • The entire algorithm is visible in one compact structure
  • The relationship of the parts of the code is quickly apparent
  • The code can be read in a straightforward top-to-bottom, left-to-right manner
  • This is almost no abstract fluff; the code is what it does, one simple step at at time
  • There is little need to visually or mentally jump around the code in the process of following it
  • There are a minimum of arguments to keep track of at each step; each function is a built-in and each here has only one or two arguments, most instantly apparent from the syntax itself, e.g. 1 // f or 1 ~f~ 2.
  • Each parameter (of stats2) is used only once, with the exception of probs; there is no interwoven handing off of arguments to track or debug (e.g. accidentally passing two in reverse order)
  • There is virtually no need to count brackets or commas

I feel that as illustrated stats2 is a sufficiently granular piece of code and that understanding and debugging it in its entirety is faster and easier than the same process on Leonid's code.

So where are the questions in all of this?

  1. Who is right here? ;^) I know that my code is faster for me to read and understand, now and later. But what do others make of it? Surely some readers are already familiar with my style (perhaps grudgingly!) -- do they find stats2 easy to read?

  2. If as I believe there should be a balance of granularity and terseness how might the optimum degree be found?

  3. Is my finding Leonid's code comparatively slow to read and follow peculiar? What methods might I employ to improve my comprehension of that style?

  4. If my code is not easy for others to read and follow how can I identify and address the barriers that make it so?

  5. Am I missing the point? Are ease and speed of reading and debugging not the primary goals of the coding style Leonid illustrated in this example? What then is, and does my style fail to meet this goal in this specific example?


Reply 1

This is a reply specifically to Leonid, not because other answers are not equally welcome and valid but because I chose his statements and code as the basis for my argument.

I suspect that there is little in this that I truly disagree with and that further dialog will bring me closer to your position. I have neither the breadth (multiple languages) nor depth (large projects, production code) of your experience.

I suspect that this is the crux of the problem: "It is somewhat an art to decide for each particular case, and this can not be decided without a bigger context / picture in mind." I think that art is what I wish to explore here.

It is somewhat unfair to pick apart your example without context but since none was provided I see no other option.

I am certainly guilty of crafting "write-only code" at times; sometimes I even find this amusing. However I do not think stats2 is a case of this. To the contrary I find it more read-friendly than your code which is largely foundation of this entire question.

I abhor code redundancy to the point of compulsively compacting other people's answers(1)(2), so your claim (if I read it correctly) that my style is inherently more redundant is simultaneously promising and exasperating. :^)

Surely I believe in code reusability, but I favor shorthand and abstractions that are broadly applicable rather than limited to a small class or number of problems. What experienced coder doesn't have a shorthand for Range @ Length @ x because that comes up frequently in a broad range of problems? But when I am going to use returnedQ again and is it worth the mental namespace to remember what it does? When I would am I going to be looking for element {0,0} again or might it be something else? Might I want Differences instead of Accumulate? Is it easier to make returnedQ sufficiently general or to simply call // MemberQ[foo] when I need it?

You wrote:

My guess is that you like terse code because it brings you to the solution most economically. But when / if you want to solve many similar problems most economically, then you will notice that, if you list all your solutions, and compare those, your terse code for all of them will contain repeated pieces which however are wired into particular solutions, and there won't be an easy way to avoid that redundancy unless you start making your code more granular.

Perhaps surprisingly this is actually rather backward from the way it is. It is easy to churn out verbose code with little thought for brevity and clarity; that is economic of my time to write. But spending the effort write terse and clear code as I attempted to do with stats2 returns economy when reading and reusing that code because I can quickly re-parse and understand this code holistically rather than getting lost in a tangle of abstractions as I do with your code. (Sorry, but that's how I feel in this case.) I should not have to run code to understand what it does; I want to simply read it in the language I am acquainted with (Mathematica).

If in the course of solving multiple related problems I realize that there is redundancy in my code I can still pull out those elements refactor my code. The simple, visibly apparent structure makes this easy.

I think the only way I shall be able to see this from your perspective is to work on a sufficiently large example where your principles become beneficial, and where our styles would initially diverge. I wonder if we can find and use such an example without pointlessly spending time on something arbitrary.


Reply 2

Your updated answer reads:

What I didn't realize was that often, when you go to even more granular code, dissecting pieces that you may initially consider inseparable, you suddenly see that your code has a hidden inner structure which can be expressed even more economically with those smaller blocks. This is what Sessions has repeatedly and profoundly demonstrated throughout his book, and it was an important lesson for me.

I welcome this epiphany! To remove redundancy from my code and make it even more terse is something I have striven for for years. I think this can only come through a direct example (or series of examples) as in the microcosm your granularity is verbose rather than condensing. How large a code base would we need to have for this level of granularity to condense code rather than expand it?

C is so verbose that I doubt I would be able to fully appreciate and internalize examples from the referenced book. Does a Mathematica-specific example come to mind?

share|improve this question
5  
From my experience Leonid's way is good habit to get. Especially if you cooperate or work for someone and accumulation of "small changes" will make it unreadable. Also, those subroutines can be reused separately, which is minimizing the whole package length. The only change I'd make is to put those definitions in reverse order and use better names. – Kuba Mar 15 at 10:41
3  
This about compositional vs holistic code. Both are with pros and cons but ultimately it's a matter of taste. As Kuba mentioned, modular code is easier to distribute, test and reuse, but of course the level of granularity must be kept below the maximum (i.e. one function for each builtin is indeed useless). Reading your code is as complicated as reading Leonid's for one who got used to a different style (again, this is subjective). I'm afraid there is little objectivity in deciding about which one is better without knowing local factors like project size, aim, how exposed it is to others, etc. – István Zachar Mar 15 at 11:19
1  
(2/2) There is a time and place for subroutines, offhand, I would say, something like manipulateThisList[list_,n_]:=list[[getPermutationsWithPropertyX[n]]] is more readable than, say list[[n//(*complicated anonymous function that does things tersely without a helper function*)]]. Sometimes, fast and efficient code cannot be written in an easy-to-understand manner. Then it can be nice to wrap it in a helper function, but the implementation of this helper function needs an explanation of the algorithm in commentary then. – LLlAMnYP Mar 15 at 11:58
3  
I'm of two minds. On one hand, I like the terse coding and employ it in my own programming. On the other hand, the method I typically employ to understanding someone else's code is to actually run it, and that requires running the steps individually and looking at the output, which is more convenient at the outset with Leonid's code. On the other other hand, picking apart the steps in Mr. Wizard's is straightforward due to the sequence of //'s (once operator precedence is understood), so this certainly isn't a dealbreaker in any sense. – march Mar 15 at 16:42
3  
You probably noted that there is this question is in the close vote queue (because it is mostly about opinions). I voted to keep this open, but more because of the quality of the question and the answers and the topic interests me than that I don't agree with the close reason. – Sjoerd C. de Vries Mar 16 at 19:32

My path to prefer granularity

This is probably more an extended comment and a complementary answer to an excellent one by Anton. What I want to say is that for a long time, I had been thinking exactly along Mr.Wizard's lines. Mathematica makes it so easy to glue transformations together (and keep them readable and understandable!), that there is a great temptation to always code like that. Going to extreme granularity may seem odd and actually wrong.

What changed my mind almost a decade ago was a tiny book by Roger Sessions called Reusable data structures for C. In particular, his treatment of linked lists, although all other things he did were also carrying that style. I was amazed by the level of granularity he advocated. By then, I've produced and / or studied several other implementations for the same things, and was sure one can't do better / easier. Well, I was wrong.

What I did realize by that time was that once you've written some code, you can search for repeated patterns and try to factor them out - and as long as you do that reasonably well, you follow the DRY principle, avoid code duplication and everything is fine. What I didn't realize was that often, when you go to even more granular code, dissecting pieces that you may initially consider inseparable, you suddenly see that your code has a hidden inner structure which can be expressed even more economically with those smaller blocks. This is what Sessions has repeatedly and profoundly demonstrated throughout his book, and it was an important lesson for me.

Since then, I started actively looking for smaller bricks in my code (in a number of languages. While I mostly answer Mathematica questions, I wrote reasonably large volumes of production code also in Java, C, javascript and Python), and more often than not, I was finding them. And almost in all cases, going more granular was advantageous, particularly in the long term, and particularly when the code you write is only a smaller part of a much larger code base.

My reasons to prefer granularity

Now, why is that? Why I think that granular code is very often a superior approach? I think, there are a few reasons. Here are some that come to mind

Conceptual advantages

  • It helps to conceptually divide code into pieces which for me make sense by themselves, and which I view as parts deserving their own mental image / name.

    More granular functions, when the split of a larger chunk of code is done correctly, represent inner "degrees for freedom" in your code. They expose the ideas behind the code, and the core elements which combine to give you a solution, more clearly.

    Sure, you can see that also in a single chunk of code, but less explicitly. In that case, you have to understand the entire code to see what is supposed to be the input for each block, just to understand how it is supposed to work. Sometimes that's Ok, but in general this is an additional mental burden. With separate functions, their signatures and names (if chosen well) help you with that.

  • It helps to separate abstraction levels. The code combined from granular pieces reads like DSL code, and allows me to grasp the semantics of what is being done easier.

    To clarify this point, I should add that when your problem is a part of a larger code base, you often don't recall it (taken separately) as clearly as when it is a stand-alone problem - simply because most of such functions solve problems which only make sense given a larger context. Smaller granular functions make it easier for me to reconstruct that context locally without reading all the big code again.

  • It is often more extensible

    This is so because I can frequently add more functionality by overloading some of the granular functions. Such extension points are just not visible / not easily possible in the terse / monolithic approach.

  • It often allows one to reveal certain (hidden) inner structure, cross-cutting concerns, and new generalization points, and this leads to significant code simplifications.

    This is particularly so when we talk not about a single function, but about several functions, forming a larger block of code. It frequently happens that when you split one of the functions into pieces, you then notice that other functions may reuse those components. This sometimes allows one to discover a new cross-cutting concern in code, which was previously hidden. Once it is discovered, one can make efforts to factor it from the rest of the code and make it fully orthogonal. Again, this is not something that is observable on the level of a single function.

  • It allows you to easily create many more combinations

    This way you can get solutions to similar (perhaps somewhat different) problems, without the need to dissect your entire code and rewrite it all. For example, if I had to change the specific way the random walk in that example was set up, I only had to change one tiny function - which I can do without thinking about the rest.

Practical advantages

  • It is easier to understand / recall after a while

    Granular code, at least for me, is easier to understand, when you come to it after a while, having forgotten the details of it. I may not remember exactly what was the idea behind the solution (well-chosen names help here), as well as which data structures were involved in each transformation (signatures help here). It also helps when you read someone else's code. Again, this is particularly true for larger code bases.

  • More granular functions are easier to test in isolation.

    You can surely do that with the parts of a single function too, but it is not as straightforward. This is particularly true if your functions live in a package and are parts of a larger code base.

  • I can better protect such code from regression bugs

    Here I mean the bugs coming from changes not propagated properly through entire code (such as changes of types / number of arguments for some functions), since I can insert argument checks and post-conditions easier. When some wrong / incomplete change is made, the code breaks in a controlled, predictable and easy-to-understand fashion. In many ways, this approach complements unit tests, code basically tests itself.

  • It makes debugging much simpler. This is true because:

    • Functions can throw inner exceptions with the detailed information where the error occurred (see also previous point)
    • I can access them easier in running code, even when they are in packages.

      This is actually often a big deal, since it is one thing to run and test a tiny function, even private one, and it is another thing to deal with a larger and convoluted function. When you work on the running code, and have no direct access to the source (such that you can easily reload an isolated function), the smaller the function is that you may want to test, the easier it is.

  • It makes creating workarounds, patches, and interactions with other code much easier. This I have experienced myself a lot.

    • Making patches and workarounds.

      It often happens that you don't have access to the source, and have to change the behavior of some block of functionality at runtime. Being able to just simply overload or Block a small function is so much better than having to overload or redefine huge pieces of code, without even knowing what you may break by doing so.

    • Integrating your functionality with code that does not have a public extension API

      The other, similar, issue is when you want to interact with some code (for example, make some of its functions work with your data types and be overloaded on them). It is good if that other code has an API designed for extensions. But if not, you may for example use UpValues to overload some of those functions. And there, having such granular functions as hooks really saves the day. In such moments, you really feel grateful for the other person who wrote their code in a granular fashion. This happened to me more than once.

Implications for larger programs

There surely isn't a single "right" way to structure code. And you may notice, that in most of the answers I post here on M SE, I do not follow the granularity principle to the extreme. One important thing to realize here is that the working mode where one solves a very particular problem is very different from the working mode when one is constructing, extending and / or maintaining larger code bases.

The whole ability to glue together things insanely fast works against you in the long term, if your code is large. This is a road to writing so-called write-only code, and for software development that is a road to hell. Perl is notorious for that - which was the reason why lots of people switched to Python from Perl despite the unquestionable power of Perl. Mathematica is similar, because it shares with Perl the property that there are typically a large number of ways to solve any given problem.

Put another way, the Mathematica language is very reusable, but that doesn't mean that it is very easy to create reusable code with it. It is easy to create the code that solves any particular problem fast, but that's not the same thing. Smaller granularity I view as an idiomatic (in Mathematica) way to improve reusability. What I wanted to stress was that reusability comes from the right separation of concerns, factoring out different pieces. It is obvious for the larger volumes of code, but I think this is no less true also for smaller functions.

When we typically solve some problem in Mathematica, we don't have reusability in mind all that much, since our context is usually confined to that particular problem. In such a case, reusability is a foreign concept and gets in the way. My guess is that you like terse code because it brings you to the solution most economically. But when / if you want to solve many similar problems most economically, then you will notice that, if you list all your solutions, and compare those, your terse code for all of them will contain repeated pieces which however are wired into particular solutions, and there won't be an easy way to avoid that redundancy unless you start making your code more granular.

My conclusions

So, this really boils down to a simple question: do you need to solve some very specific problem, or do you want to construct a set of bricks to solve many similar problems. It is somewhat an art to decide for each particular case, and this can not be decided without a bigger context / picture in mind. If you are sure that you just need to solve a particular problem, then going to extreme granularity is probably an overkill. If you anticipate many similar problems, then granularity offers advantages.

It so happens that large code bases frequently automate a lot of similar things, rather than solve a single large problem. This is true even for programs like compilers, which do solve a single large problem, but in reality lots of sub-problems will reuse the same core set of data structures. So, I was particularly advocating granularity in the context of development of large programs - and I would agree that for solving some particular very specific problem, making it too granular might result in too much of a mental overhead. Of course, that also greatly depends on personal habits - mine have been heavily influenced in recent years by dealing with larger chunks of code.

share|improve this answer
1  
I added a link to the book. Hopefully, it is the correct one. – rcollyer Mar 15 at 20:52
    
@rcollyer Thanks, this is it! – Leonid Shifrin Mar 15 at 21:09
    
A couple of half-baked responses are now included in my question. – Mr.Wizard Mar 15 at 22:53
    
@Mr.Wizard 1. I will refrain from adding replies to your replies to my answer, since I think that SE doesn't have the right format for this sort of discussions, but will try to briefly reply in comments here. For your first reply: I think the truth (for me at least) is somewhere in the middle. Not all my code looks as granular as presented in that example - in some sense, it was an intentional exaggeration to illustrate the point. In practice, the degree of granularity I use is usually mostly determined by the DRY principle - so that no essential blocks of code are repeated, even small ones. – Leonid Shifrin Mar 16 at 14:59
    
@Mr.Wizard 2. Granularity helps me on several levels, some of which I already mentioned. a. It helps to conceptually divide code into pieces which for me make sense by themselves, and which I view as parts deserving their own mental image / name. b. It helps to better understand the code after a while, because I may not remember exactly what was the idea behind the solution (well-chosen names help here), as well as which data structures were involved in each transformation (signatures help here). c. It helps to separate abstraction levels: the code combined from granular pieces reads like ... – Leonid Shifrin Mar 16 at 15:03

This is a nicely formulated and stimulating question. The answer of course is "Mu". The comment of István Zachar is among those lines... (Here is the official "Mu" link.)

Usually when we cannot answer a question in the terms it is asked we extend the paradigm. (E.g. coming up with complex numbers. Or getting enlightened. Or bringing straw men.) So, what follows is my view of how the opposed in the question perspectives can be resolved. For simplicity of the explanations I both over-simplify and over-exaggerate.

The names "Leonid" and "Mr. Wizard" are used to designate code producing agents as described in the question. Those names should not be seen as literal references to the programming activities of the corresponding persons.

Diagnosing the dichotomy

  1. In Object-Oriented Programming (OOP) languages the key concept is reuse. We do not want to have the domino effect when changing parts of the code because of new understanding of the problem domain the code operates in. That is why we encapsulate behavior in objects, etc. We can say that modeling in OOP is with an entity centric language: the entities are objects and they exchange messages. Leonid's breakdown of functions can be seen to come from similar considerations (with a structural programming flavor).

  2. In Functional Programming (FP) languages we have the ability -- and we are enticed -- to refine our code iteratively until we derive essentially a new language that fits the problem domain of our code. FP provides modeling with a verb centric language. The valency of the verbs can be used to make new verbs, etc. Fundamental understanding of the problem domain brings the derivation of new verbs. Mr. Wizard's coding style comes from that perspective.

  3. A person would prefer one or the other approach based on person's native language and scientific, engineering, or mathematical background and the audience. For example, chemists would prefer OOP (and flow charts) because they study processes of matter changing states. A mathematician might select different styles when communicating through code to (i) other mathematicians in academia, or (ii) software engineers in large companies.

  4. The two approaches should not be just contrasted but seen as dual. For example, consider a graph G in which the nodes are entities and the edges are messages exchanged by the entities. This graph corresponds to OOP. Now consider the line graph L(G) in which the edges of G become nodes and the nodes of G become edges. That L(G) graph corresponds to FP.

  5. Leonid does FP by using L(G) i.e. by seeing functions as objects. Mr. Wizard does FP using G but seeing edge paths not vertices and links between them.

Proposed cure

One way to address the opposing forces of the two perspectives is to factor out the reuse concern from the brevity and simplicity of implementation concern.

One way do this is to have a Domain Specific Language (DSL) that is easy to use and understand. Say, a DSL close to the natural language of the intended users. The DSL will bring what Leonid wants, the interpreter of the DSL would bring code what Mr. Wizard wants. (And if done right Mr. Wizard might enjoy reading it.)

This can be facilitated with different tools. Below I demonstrate the use of Functional Parsers and the OOP Design Pattern Interpreter.

The code in this answer of mine illustrates fairly well "a DSL close to the natural language of the intended users" for programming tasks.

Example

Below is (another) simple example that shows separated and reusable formulation in Leonid's manner, and coding in Mr. Wizard's manner. (Also see this answer of mine about DSLs design and application in Mathematica.)

  • The coding style of Leonid is used to come up with the EBNF grammar.

  • The coding style of Mr. Wizard is used in both the EBNF grammar rules formulation and under the hood of the parser generation implementation. For the former see the functions applied at the end of some of the grammar rules, after the characters "<@". For the latter see the article "Functional Parsers" by Fokker -- it is exactly in the style Mr. Wizard likes. (My implementation FunctionalParsers.m is messier since it was written with an older Mathematica version.)

enter image description here

To be clear, this example shows that the two approaches can be both used by a higher abstraction layer. Mr. Wizard's approach can be tucked in into an interpreter and Leonid's approach would require users to program with clear natural language commands that can be decomposed into/within context free grammars. Mr. Wizard's approach facilitates the parsing and interpretation of those grammars, but otherwise is not visible to the users.

Conclusion

From the above it is clear that with the right attitude, automated tools, and smart computers Mr. Wizard's perspective becomes obsolete to humans, and Leonid's perspective endures, it bubbles up to the end users.

(Sorry if this conclusion seems too pessimistic. In my defense we increasingly live in a world that is all about outsourcing, offshoring, automation, and "the robots are coming"...)

share|improve this answer
    
Anton, I just woke up and I am probably thinking even less clearly than usual, but I am quite lost by your conclusion. I think I follow most of the numbered points and the graph analogy is very interesting. However I am not sure what your example represents, and I don't see how your conclusion that "Mr. Wizard's perspective becomes obsolete to humans" follows from it. (continued) – Mr.Wizard Mar 15 at 21:01
    
Until I hit your conclusion I was thinking that perhaps it shows that one could code in a natural language DSL (command) that is close in structure to what I visualized as I wrote stats2, and have it parsed into a verbose (granular?) form automatically. But now I think not? – Mr.Wizard Mar 15 at 21:05
    
Just to be clear, there are parts of code where I don't use the "extreme granularity" approach - usually those are functions which implement certain "engines" (however small), and where parts interact too tightly (e.g. are entangled by complicated scoping structure) so that decoupling them will bring more harm than good. One such example was a dtraverse function in the same post (although even there I tried to make it as granular as possible), another is e.g. withRedefined function from this post. @Mr.Wizard – Leonid Shifrin Mar 15 at 21:56
2  
@LeonidShifrin I have to make a disclaimer in my post that the use of your and Mr. Wizard names is some sort of level of indirection of the programming styles presented in question. It should not be seen as literal references to your programming activities. – Anton Antonov Mar 15 at 22:03
1  
@AntonAntonov Actually I will probably add an example of a DSL to one of my posts on large code. Personally I am very enthusiastic about DSLs, and think that this is a very underused area in the context of Mathematica programming. Surely, there are internal DSLs in Mathematica, but this technique has not been promoted as an effective programming technique to the end users. – Leonid Shifrin Mar 15 at 22:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.