Once more on object orientation in Mathematica: does it have to be so hard?

Recently I came across a set of problems which would be solved most easily within an object-oriented approach. I first attempted to solve them by other means, but found the complexity of the code growing too fast and, while individual functions used a mixture of styles, the overall feel and organization of the code was de-facto procedural. Having looked at the final code, I saw that I would have written it much faster and cleaner in Java, which is when I realized that OO was needed. Basically, I needed it to reuse some behavior and decrease coupling between components.

We all know that there were a number of attempts on implementing OO extensions for Mathematica. However, none of them IMO were very simple, and none of those I looked at had a natural "look and feel" for Mathematica (I could have missed some). OTOH, we also know that Mathematica has powerful metaprogramming capabilities, which should make it possible to implement the core of an OO extension rather easily.

Since OO is a broad paradigm, here are some requirements to restrict the problem. The desired OO extension should

• Be idiomatic. Programming in it should feel natural for experienced Mathematica users. In particular, it should go well with immutable Mathematica expressions, and have the least possible problems concerning interoperability with Mathematica (garbage collection - related problems etc).

• Support instantiation, inheritance and polymorphism. By inheritance I mean reuse of behavior (methods) rather than state (fields).

• Reuse as much of the Mathematica's core constructs as possible

• Limit the things that can be done with it as little as possible, compared to the programming techniques and tricks we are used to in Mathematica

• If possible, have at least minimally convenient syntax for things like method calls.

• Implementation should be simple in the sense that it should not introduce too many exterior constructs, such as ToString - ToExpression cycles, needless manipulations with contexts, creation of new symbols which are not strictly necessary, etc.

I am not so much worried about efficiency here, since the intended purpose for such an extension is more to help with larger-scale code organization for certain classes of tasks, than to be used with millions of light-weight objects.

In many previous questions on similar topics, the two rather separate topics were often mixed together. One concerns mutable struct-like data structures and their possible implementations in Mathematica, and the other was about OO proper. So, to clarify a bit more, I am not so much interested in the former topic here. I am rather more interested in the dynamic aspects (behavior reuse), and ways to conveniently package code and organize larger projects, but not so much in efficient mutable data structures. In other words, the subject matter is what can the object orientation bring to the table for larger Mathematica projects, and how can we make the most out of it in Mathematica.

So, the question is: can we have a core of such an OO extension in under 100 lines of Mathematica code?

-
@NasserM.Abbasi It is not free for one thing, and I really want a simple and short implementation for another one. –  Leonid Shifrin Dec 23 '12 at 15:15
@NasserM.Abbasi This is not just on V9 - the form I used in undocumented. But it is the form I needed, and there isn't an easy replacement for it within the documented functionality. –  Leonid Shifrin Dec 23 '12 at 15:34
Would you consider expanding this into a blog post eventually, after you've gotten some feedback? –  Szabolcs Dec 25 '12 at 1:08
Leonid, you are a modern marvel! :) Lately you seem to have been producing new and interesting projects at an incredible pace. I have little practical familiarity with the OO paradigm, but despite frequent calls for a Mathematica implementation I had tended to think it was probably not very useful in this context. That you hold the opposite view has surely made me think again, and I'm very much looking forward to having the time to give this the consideration it deserves! –  Oleksandr R. Dec 25 '12 at 3:38
@Szabolcs Yes, absolutely. That's the plan, actually. I know that it may seem like I've forgotten all my promises regarding the blog, but this isn't so. And for this topic, I posted this here since I wanted to attract more attention to this topic and make the core of this more accessible to anyone. I have much more stuff regarding OO, and that stuff will go into the blog post. For example, here I wasn't able to consider more meaningful / complex examples of applying this stuff to practical problems, but the blog will give me such an opportunity. –  Leonid Shifrin Dec 25 '12 at 7:47

The answer seems to be yes. At least, I will try to describe an attempt which would pass the "under 100 lines of code" test. How well it satisfies the other criteria is a subjective matter, but I have already used it for larger-scale project with so far very positive results.

I will present two implementations. One is an absolute toy, but extremely simple. The other is also a toy, but having all the core requested features. The one I now use in real project is somewhat more complex, but has the same core.

A toy implementation in under 25 lines of code

Implementation

Here is the toy implementation. It is small enough that I can present it first and explain afterwards. The following implements the type declaration operator, where the type is represented by a symbol:

ClearAll[DeclareType];
DeclareType[s_Symbol] :=
Function[Null,
ClearAll[s];
SetAttributes[s, HoldAll];
defineMethods[s, ##];
s,
HoldAll];


This does not do much until defineMethods has been defined. Neither does this support inheritance, so we add the following:

DeclareType[s_Symbol ~ Extends ~ p_Symbol] :=
Function[Null,
DeclareType[s][##];
SuperType[s] = p;
s[self_][lhs_] := p[self][lhs];
s,
HoldAll];


Here, Extends is an inert head introduced for clarity only, and SuperType holds a value of the supertype, if there is any. Finally,

ClearAll[defineMethods];
SetAttributes[defineMethods, HoldRest];
defineMethods[s_] := s;
defineMethods[s_, SetDelayed[lhs_, rhs_], rest___] :=
Module[{},
s[cont_][lhs] :=
Block[{$self = s[cont],$super = SuperType[s][cont]},
rhs
];
defineMethods[s, rest]];


This is all there is here. This introduces two special symbols $self and $super, which can be used inside the body of any method defined for a type, to refer to the object itself or call methods of its supertype. The way this works is as follows: I use rules to define methods as SubValues for the type symbol, and the patterns used in the method definitions naturally become the parts of the resulting SubValues.

Examples

This declares the type Animal:

DeclareType[Animal][
breathe[] := Print["I am breathing ", $self[[1, 1]]], sleep[duration_String: "one hour"] := "I have been sleeping for " <> duration, sleep[duration_Integer] :=$self@sleep[ToString[duration] <> " hours"],
move[] := Print["I move ", $self[[1,2]]] ] (* Animal *)  Now we create an instance of this type: an = Animal[{"oxygen","fast"}] (* Animal[{oxygen,fast}] *)  Note that the instance is completely stateless, it is fully defined by an expression (list) containing its content. We can now call methods: an@breathe[] During evaluation of In[68]:= I am breathing oxygen an@sleep[] (* I have been sleeping for one hour *) an@sleep["two hours"] (* "I have been sleeping for two hours" *) an@sleep[5] (* "I have been sleeping for 5 hours" *)  Already here, we can note one very important big advantage of this scheme: one can use the usual sweet Mathematica pattern-matching to define methods, including all (well, most of) the nice stuff, such as default values, overloading, etc. But what about inheritance? Well, here is an example: DeclareType[Cat ~ Extends ~ Animal][ sleep[] := StringJoin[$super@sleep[], $self[[1, 3]]] ] (* Cat *)  We now create an instance: cat = Cat[{"oxygen","fast"," on the floor"}] (* Cat[{oxygen,fast, on the floor}] *)  and call the methods: cat@breathe[] During evaluation of In[125]:= I am breathing oxygen cat@sleep[] (* I have been sleeping for one hour on the floor *) cat@sleep["three hours"] (* I have been sleeping for three hours *)  Note that inheritance works as one would expect. In particular, the Cat's version of sleep method is called for zero arguments, while Animal's version is called for other calls for sleep. Note also that the Cat's sleep method has access to the parent's method via $super.

In a sense, this is probably as close to an ideal mix of OO and Mathematica as it gets: objects are completely stateless and based on immutable Mathematica expressions. However, this approach has a number of flaws. In particular, it requires that all the symbols used as method names do not have definitions such that they can prematurely evaluate. For example, having defined something like

breathe[] := Print["breathe"]


would spoil the above behavior. It is rather desirable that things be more robust. Another problem (which would persist to my other implementations) is shadowing. If two different users define types with the same method names which live in different contexts, shadowing will happen. It could probably be solved by converting symbol names to strings and then using only the short symbol names, but I think this is not an idiomatic Mathematica solution, in many ways. So, the other way to solve this (and this is what I do) is by convention: introduce a special context (e.g. OOMethods), and place all symbols used for method names there. This can also solve the evaluation problem, if in addition one adopts a convention to not assign any rules to those symbols. The above declaration for Animal would then look like:

DeclareType[Animal][
OOMethodsbreathe[] := Print["I am breathing ", $self[[1, 1]]], OOMethodssleep[duration_String: "one hour"] := "I have been sleeping for " <> duration, OOMethodssleep[duration_Integer] :=$self@OOMethodssleep[ToString[duration] <> " hours"],
OOMethodsmove[] := Print["I move ", $self[[1,2]]] ]  Note that the user of the type does not have to use long names, as long as both the type's context and OOMethods are on the $ContextPath.

A more real thing

The implementation I am going to show now is the core of what I ended up using, but is based on the same set of ideas. The main technical difference is that, in order to prevent premature evaluation of method calls, I found no other way than to introduce some state into the objects / instances. They will be now represented by unique (within a given Mathematica session, but this restriction can be lifted) symbols. This is similar in spirit to how JLink implements references to Java objects. And I will use UpValues as a tool to attach both state and behavior to these symbols. While this scheme would help to solve the premature evaluation problem I mentioned, it also opens additional possibilities not easily possible in the simple setting of the previous section, such as attaching new methods to a single instance(rather than a type) at run-time. This by itself is a powerful enough feature to justify this approach (it is not supported by e.g. Java, but is supported by e.g. Javascript).

So, without further due, here is the new DeclareType:

ClearAll[DeclareType];
DeclareType[Object] = Null;
DeclareType[type_Symbol] := DeclareType[type~ Extends ~ Object];
DeclareType[Object ~ Extends ~ _] = Null;
DeclareType[type_Symbol ~ Extends ~ superType_Symbol] :=
Function[
Null
,
ClearAll[type];
If[ValueQ[vtable[type]], vtable[type] =.];
SetAttributes[type, HoldAll];
defineMethods[type, ##];
SuperType[type] = superType;
type
,
HoldAll
];


It is not so much different from what we had before. The new symbols it relies on are defined as:

ClearAll[Object, object];
Object::nomethod = "Unknown method for type Object. The method call was 1";
SetAttributes[{Object, object}, HoldAll];
Object[__] := object;
object[args___,methodCall_] :=
(
Message[Object::nomethod,ToString@HoldForm[methodCall]];
$Failed );  (this is similar in spirit to Java Object - so that there is a single object hierarchy). Also, SuperType is now defined as ClearAll[SuperType]; SuperType[Object] = Null; SuperType[_] = Object;  A symbol vtable stores a reference to a dispatch function for any type. It is initially defined as ClearAll[vtable]; vtable[Object]:=object;  but is dynamically acquires new definitions as new types are being defined. Here is a new defineMethods, and this is where things start to get substantially different: ClearAll[defineMethods]; SetAttributes[defineMethods, HoldRest]; defineMethods[s_] := s; defineMethods[s_, args___] := Module[{ff}, SetAttributes[{ff}, HoldAll]; vtable[s] = ff; s[content_] := Module[{sym}, SetAttributes[sym, HoldAll]; sym /: instanceValue[sym] = content; sym /: Normal[sym] := With[{cont = instanceValue[sym]}, HoldForm[s[cont]] ]; (* Note: this forces the arguments to be evaluated *) sym[f_[argums___]] := Hold[f][argums]/.Hold[h_][x___]:> ff[sym,f[x]]; sym /: Set[sym, newcontent_] := sym /: instanceValue[sym] = newcontent; sym/: TypeOf[sym] = s; sym ]; addMethods[args][s]; ];  Here, the input symbol s stands for the name of the type. What happens here is that every time I call SomeType[some-content], I get a new symbol with the content attached via UpValues and the instanceValue container, and the behavior attached via DownValues and the ff symbol, which is a dispatch function for a given type. The latter is defined only once per type definition, and is stored in vtable[SomeType]. Also, the TypeOf operation is defined (similar to Java getClass), and one can also replace the full content of the object using Set. The instanceValue container is defined as ClearAll[instanceValue]; SetAttributes[instanceValue,HoldAll]; instanceValue[_]:= Throw[$Failed, {instanceValue,"invalid_instance"}];


The final piece we need is addMethods. Here is the code:

ClearAll[addMethods];
With[{ff = vtable[s]},
ff[sym_, lhs] :=
Block[{
$self = sym,$content = instanceValue[sym],
$super } , SetAttributes[$super,HoldAll];
$content /: Set[$content, expr_] := Set[sym, expr];
$super[call_]:= vtable[SuperType[s]][sym,call]; rhs ]; addMethods[rest][s] ]; addMethods[][s_] := vtable[s][sym_, lhs_] := vtable[SuperType[s]][sym,lhs];  What happens here is that the definitions for the type's dispatch function ff are formed based on the definitions of the methods, supplied by the user. Again, we win big due to the generality of patterns, which allows us to automatically create valid definitions for ff by doing a scope surgery on the original definitions of the methods. Another thing worth noting is again the user of Block which allows the user to use symbols $self, $super and $content inside a body of a method.

The same examples as before should work here. In one piece, the code lives here. I left out some more advanced things such as formatting, per-instance method addition at run-time, automatic object release (when objects are no longer needed), etc. These are all implemented in the complete full version that lives here. In particular, the full version has a very simple installation procedure described in the README file in that linked gist. It has an example notebook, which is opened by a simple command GetExampleNotebook[], once the project has been installed and loaded.

Use cases and limitations

The OO extension described above is intended to be used as a tool for large-scale project organization. In this capacity, I tested it already on real projects and so far have had very positive results, in terms of code complexity management, modularization of component, and making code more reusable.

Some real use cases

Here I will list some real examples which rely on it:

I will add more documentation to these, so that it will be easier to explore them, but in fact they are very easy to use.

The code of these example projects also shows how one can do practical things, such as attaching or changing the state of an object. The basic idea is that it is better to avoid adding explicit state (in fact, the current version of OO does not support object fields, only object methods. This was intentional, but such a support can be easily added if there will be a consensus that they are needed). Instead, one can add new methods which would encapsulate the state. This allows to avoid introducing mutable state in objects in many cases, which arguably leads to a cleaner code.

Limitations

This OO extension has not been designed with high-performance applications in mind. While you can create thousands of objects and use it also for lightweight objects, you will face more or less the usual high-level Mathematica overhead (in my view, this is not so much of a real limitation, since Mathematica has other means for high-performance programming, and OO is often not really needed on such a "fine-grained" scale). Also, one must be careful to avoid the memory leaks.

Certain types of patterns in method definitions may also not work, in particular conditional patterns where condition is attached to the left-hand side, such as

myMethod[x_,y_]/;x<y:=body


Again, this limitation can be rather easily removed by making a slightly more complex parser for addMethods. OTOH, the definitions like

myMethod[x_,y_]:=body/;x<y


should work, so this is perhaps not so much of an issue.

Further developments

While I feel that the current state of OO is minimally complete, in the sense that it can be used in serious applications, I plan some further developments, such as

• Reflection a-la Java (meta-information on methods, etc)
• Integrated unit-tests
• Some other enhancements

The extension was initially build not just for its own sake, but to provide a base for the development of other projects of interest to me (and very possibly also to a wider audience). So I expect that it will be further shaped by the needs of those projects people build using it. So, any feedback on the system is much appreciated, and will affect further developments.

-
Strap in the for ride: here we go again! :-D –  Mr.Wizard Dec 23 '12 at 15:30
@Mr.Wizard I have to go now, but will do it later today. –  Leonid Shifrin Dec 23 '12 at 15:47
Could sym[f_[argums___]] := Hold[f][argums]/.Hold[h_][x___]:> ff[sym,f[x]]; be replaced with sym[f_[argums___]] := {argums} /. {x___} :> ff[sym,f[x]];? –  Mr.Wizard Dec 23 '12 at 15:51
@Mr.Wizard Probably yes. I guess the first one was a remnant from some original code, and I did not rethink it after changes. Thanks, I will edit this in soon. –  Leonid Shifrin Dec 23 '12 at 15:54
I suppose sym[f_[argums___]] := ff[sym,f[##]] &[argums]; might work too. –  Mr.Wizard Dec 23 '12 at 17:20
show 1 more comment