I think after six months of exposure to Mathematica and the Wolfram Language I am fairly OK with writing short codes and snippets. However, what are some general strategies to use in order to write big blocks of code?
I had a talk devoted specifically to this topic, on Second Russian WTC in 2014. Unfortunately, it is in Russian. But I will try to summarize it here.
Since this post is becoming too long, I decided to split it to several smaller ones, each dedicated to some particular set of methods / techniques. This one will contain a general / conceptual overview. As I add more specific parts, the links to them will be added right below this line.
From the bird's eye perspective, here is a list of problems typically associated with the large-scale development, which is largely language-agnostic:
Some of the well-known methods to tame projects complexity include:
All these methods basically help to reach a single goal: improve the modularity of the code. Modularity is the only way to reduce complexity.
To improve modularity, one usually tries to:
Here I will list some general techniques, which are largely language - agnostic, but which work perfectly well in Mathematica.
Embrace functional programming and immutability
A lot of problems with large code bases happen when the code is written is stateful style, and state gets mixed with behavior. This makes it hard to test and debug separate parts of the code in isolation, since they become dependent on the global state of the system.
Functional programming offers an alternative: program evaluation becomes a series of function applications, where functions transform immutable data structures. The difference in resulting code complexity becomes qualitative and truly dramatic, when this principle is followed down to the smallest pieces of code. The key reason for this is that purely functional code is much more composable, and thus much easier to take apart, change and evolve. To quote John Hughes ("Why functional programming matters"),
I actually highly recommend to read the entire article.
In Mathematica, the preferred programming paradigms, for which the language is optimized, are rule-based and functional. So, the sooner one stops using imperative procedural programming and moves to functional programming in Mathematica, the better off one will be.
Separate interfaces and implementations
This has many faces. Using package and contexts is just one, and rather heavy, way to do that. There exist also ways to do that on the smaller scale, such as
Master scoping constructs and enforce encapsulation
Mastering scoping is essential for scaling to larger code bases. Scoping provide a mechanism for information - hiding and encapsulation. This is essential for reducing the complexity of the code. In non-trivial cases, it is quite often that, to achieve the right code structure, even inside a single function, one may need three, four or even more levels of nesting of various scoping constructs (
Separate orthogonal components in your code
This is a very important technique. It often requires certain advanced abstractions, such as higher-order functions and closures. Also, it requires some experience and certain way of thinking, because frequently code doesn't look like it can be factored - because for that certain parts of it should be rewritten in a more general way, yet it can be done. I will give one example of this below, in the section on higher-order functions.
Use powerful abstractions
Here I will list a few which are particularly useful
Use effective error-reporting in internal code, make your code self-debugging
There are a number of ways to achieve that, such as
All them combined, lead to a much simpler error diagnostics and debugging, and also greatly reduce regression bugs
Use unit tests
There has been enough said about the usefulness of unit tests. I just want to stress a few additional things.
Topics not covered yet (work in progress)
To avoid making this post completely unreadable, I did not cover a number of topics which logically belong here. Here is an incomplete list of those:
There are a number of techniques which may be used to improve the control over code bases as they grow larger. I tried to list a few of them and give some examples to illustrate their utility. These techniques can be roughly divided into a few (overlapping) groups:
This is surely not an ideal classification. I will try to make this post a work in progress and refine it in the future. Comments and suggestions more than welcome!
Here are some advices from my experience.
Managing the complexity, II: controlling complexity on the smaller scale
There are a few things you can do to control and reduce the complexity of your code, even on the small scale - long before you move to packages and split code into several files.
Effective use of the core data structures
This is probably the first thing to mention. The most important core data structures are
Some of the properties which make both
However, in the long term, one has to be aware of certain flaws as well:
In most cases, it is much better to split your code in a number of fairly small functions, each one doing some very specific task. Here are a few suggestions regarding that:
Example: simplistic DOM viewer
Below is the code of a rudimentary viewer for a DOM structure of an HTML page:
You can call
This is what I call granular code: it contains a few really tiny functions, which are very easy to understand and debug.
Example: modeling and visualizing random walks
This one was a real question asked by someone in the Russian-speaking Mathematica online group. It is valuable since this is a real problem, and it was originally formulated in a procedural style.
The problem is to model a 2-dimensional random walk with certain step probabilities, which are constants (don't depend on the previous steps). The question asked is to find a probability to return to the point of origin in less than a given number of steps. This is done using essentially Monte-Carlo simulation, running the single walk simulation multiple times, and finding how many steps it took to return back, for a particular experiment.
Here is the original code. The problem settings (I keep the original code):
The actual computation
Here is the visualization of the experiment (basically, the unnormalized empirical CDF and PDF):
The original complaint was that the code was slow, and the asker was looking for ways to speed it up. Here is my version of the code, rewritten in a functional granular style:
We run it as
It ends up 10 times faster, but also the above code is, at least for me, much more readable - and you can easily test all individual functions, since they don't depend on anything that has not been passed to them explicitly.
Now, here is a version of the same code, expressed as a one-liner:
What I would say is that I strongly prefer the granular version, in all cases but those where the condensed one offers far superior performance, and only if this is critical for the problem. In this particular case, the performance is the same, and in most other cases it also won't be worth it to keep such code, since it is much harder to understand.
In any case, to me this example serves as another good illustration of the advantages and superiority of functional programming done in a granular fashion, and I hope it additionally illustrates my point about the importance of granularity.
Writing code in this style is very beneficial for readability, extensibility and the ease of debugging. Do it, when you can.
Example: inverting many to many relationships
I will borrow this one from this answer. The function below inverts many-to-many relationship encoded in an association:
Here is an example of use:
But here I just want to stress the way the function is written: using
When you define functions using patterns, you can use function overloading - giving several definitions to a single function, on various number / types of arguments. Languages which support overloading, have mechanisms for automatic dispatch to the right definition, given specific input arguments. This automation can be used to simplify programmer's life and write more expressive code. Mathematica fully supports overloading via its core pattern-matching engine, and in fact its pattern-matching capabilities can be thought of as "overloading on steroids" in this context, compared to other languages - even those which support multiple dispatch. You can actually often design your code in a such a way as to maximally utilize this option.
Functions written in such a style are typically (not always though) much more readable and extensible, than if you would have a single large
I will illustrate this with a single example taken from the RLink module source code: this single function determines the type of all RLink objects, either sent to R from Mathematica, or received from R:
This example illustrates two more quite useful tricks: use local variables shared between the body and the condition of the rule, and use the catch-all pattern to throw local (internal) exception - but these I will discuss separately.
To summarize advantages of this method:
Some of the things to watch for are, though:
Small scale encapsulation: inner functions
This is a form of encapsulation, where you introduce inner functions, local to the
Example: directory traversals with skips
Here is an example I took from this Mathgroup post and modified:
The usage is:
Before we run this code, a few words about it. It is all built on inner functions and closures. Note that all of the
The code structure here is very transparent. I view nested directory traversal with functions
Let us now see what the run-time skip facility can give us. Here I will run through the entire directory tree for the
Now I do the same, but instruct the code to skip the inner sub-directories' trees, setting
And get the same result, only 1000 times faster. I think this is pretty cool given that it only took a dozen lines of code to implement that. And using inner functions and closures made the code clear and modular even on such a small scale, and allowed to cleanly separate state and behavior.
Example: Peter Norvig's spelling corrector in Mathematica
In the following example this idea is pushed to the extreme. Here is where it comes from. It is hard to beat the clarity and expressiveness of Python, but at least I gave it a try.
Here is a training data (it takes some time to load this):
Here is the code I ended up with (I cheated a bit by abbreviating a number of built-ins using
Here are some tests:
The above code illustrates another thing about inner functions: you may use them also to significantly change the way the code looks inside
I personally use inner functions all the time, and consider them an important tool for improving small-scale encapsulation, structure, readability and robustness of the code.
One thing to watch out for is that in some cases, inner functions are not garbage - collected automatically. This usually happens when some external objects point to them at the time when they are defined. This may or may not be acceptable, depending on your circumstances. There are also ways to avoid it, such as using pure functions (which, however, can't be easily overloaded and are generally less expressive since you can't easily do pattern-based arguments destructuring and tests for them).
Final example: Huffman encoding
To illustrate many of the points I mentioned above, I will here provide my re-implementation of the Huffman encoding algorithm, based on the code from David Wagner's excellent book. So I refer to his exposition for details on the algorithm and ideas used. I rewrote it to use
We start with the test message, same as in Wagner's book:
Building Huffman tree
Here is all the code needed to build Huffman tree from an arbitrary list of elements:
Here is the tree in our case:
Here is all the code needed to encode the message, given a Huffman tree:
Now we encode our message:
Here is the code to decode the message:
So that we have
I think, this example illustrates very well the kind of economy and simplicity that is possible to get from a combination of functional programming, very granular code, function overloading, function composition, operator forms / currying (note that I actually introduced currying / operator form also for the user-defined
All code contains absolutely no mutable state. Except for inner functions, it uses all of the techniques I described above. The result is a tiny program that solves a non-trivial problem, and while there is no room here for code dissection, it is very easy to take this code apart and understand what goes on at each step. In fact, it is mostly clear how it works just from looking at the code.
Of course, the main credit goes to David Wagner, I just made a few changes to utilize a few recent additions like Associations and completely remove any mutable state.
Managing the complexity III: using powerful abstractions
In this section I will list a few techniques which allow one to write more modular code and better separate the concerns, by using certain powerful abstractions provided by or possible to have in the Mathematica.
These are functions which take other functions as arguments. In Mathematica, a number of core built-in functions like
The utility of this construct can be seen most clearly within the functional programming paradigm. Higher-order functions can be used to parametrize generic functionality, where custom behavior is injected with functional arguments. This allows one to easily separate generic functionality from the specific.
This answer is based on the original poster's statement that he has been using Mathematica for six months, and is now trying to build something a bit more complex. I do not take this to mean a large project in the sense that an application developer would use the word.
The notebook interface is really easy to experiment in, and I know that when I had used Mathematica for six months I still needed to test my code very often to make sure it worked. So I used to have a strong preference for the notebook interface. Now I can write much larger pieces of code and be reasonably sure that they will work, so I'm warming up more to IntelliJ which I use sometimes, especially when I am authoring a package.
Unfortunately trying to use both simultaneously - writing on a package in IntelliJ and experimenting with aspects of the package in Mathematica - creates namespace issues that I am usually not willing to fight against, although I have done that too on occasion.
Packages solve mainly three problems in my opinion
If none of these is a concern of yours then you are probably better off writing a notebook in my experience. Note that your mileage with a notebook depends on how you structure it, for larger projects (but not large enough for an IDE) I make use of titles, subtitles and text a lot.
If you find that you have a large code block in a single function then you are probably not writing good, functional code. When you write functional code you want to build up your application gradually using only small functions. The smaller a function is the more testable it is, and this is very important when you are working on a larger project. Otherwise you will find yourself in a situation where you are progressing very slowly because it's hard to debug problems that crop up.
Here is an example of a project with fifty or so functions, but because they have been nested into sensible categories they are all easy to find:
Use an IDE like Workbench and remember good software development practices
Take a look at workbench which is a branded version of Eclipse, a very common Integrated Development Platfom (IDE). On the WRI site a lot of information is given of which I would like to point you to a White Paper on Building Large Software in Mathematica. It contains very helpful advice on organizing around packages, using unit tests, version control and a lot more:
"Nothing is more practical than a good theory" (Kurt Lewin)
Many people point at Roman Maeder's book The Mathematica Programmer and I would also add his other book, Computer Science with Mathematica, which I have found to give excellent theoretical background about general principles of programming rather than simply listing some techniques (e.g. know why).
Most helpful there -- and not too commonly found in other books to my knowledge -- are theses topics:
For large development projects it is a good idea to use the well established, understood, and documented Object-Oriented Design Patterns as explained and exemplified in my presentation "Object Oriented Design Patterns" at the Wolfram Technology Conference 2015. (The presentation recording is also uploaded at YouTube.)
This diagram shows the large context of patterns:
Here is a link to a document describing how to implement OOP Design Patterns in Mathematica:
The document has almost all of the material covered in my presentation linked above. Of course it has some additional material.
Comparison of Functional Programming and Design Patterns implementations
This MSE answer of mine provides a discussion and links to two implementations of GitHub repository commit history visualization. One of the implementations is with functional programming, the other with Design Patterns. The videos and the document linked above use for examples the package GitHubDataObjects.m from that answer.
As Leonid mentions in one of his answers one of the methods of managing complexity is using Domain Specific Languages (DSLs). In this answer I will provide links to documents, packages, blog posts, and discussions of creating and utilizing DSLs in Mathematica.
For a 2.5 minutes introduction see this video between 25:00 and 27:30.
When to apply DSLs
Here are some situations for applying DSLs.
See the code example below illustrating steps 3-6.
Introduction to using DSLs in Mathematica
The blog post "Simple time series conversational engine" discusses the creation (design and programming) of a simple conversational engine for time series analysis (data loading, finding outliers and trends.)
Here is a movie demonstrating that conversational engine: http://youtu.be/wlZ5ANglVI4.
This example is for the steps 3-6 of the second section.
Load the package:
Give an EBNF description of a DSL for food craviings:
Generate parses from EBNF string:
Test the parser
Note the EBNF rule wrappers -- those are symbols specified at the ends of some of the rules.
Next we implement interpreters. I am using
Here the parsing tests are done by changing the definitions of the wrapping symbols