# Strategies for creating links to external systems

Often I find myself creating links to external systems, be it a converter from Mathematica to python, a link to Git/GitHub or to an electronic structure package.

And I am far from the only one. I've seen links to MATLAB, to POVRay, WordPress, etc.

Each time I do this I find myself using a different method, so I wanted to get some strategies / best practices.

How do other people do this?

• I think this is much too broad. The examples you gave are quite different. Even if the question were restricted to creating interfaces to other programming languages (much more specific), it would be very broad. But then at least I could imagine an equally unspecific answer based on the differences between approaches in e.g. J/Link, MATLink, RLink, ExternalEvaluate. – Szabolcs Feb 25 '18 at 12:35
• @Szabolcs I would agree. Even within links to languages, I don't think there is a single common pattern - the specific strategy is determined by a number of factors, such as dominant programming paradigm of the other language, the impedance mismatch between that language and Mathematica, the set of intended use cases, performance requirements, etc. – Leonid Shifrin Feb 25 '18 at 21:40
• @Szabolcs Besides, the required amount of work to implement such strategies can be vastly different for different ones. For example, my estimate is that it is an order of magnitude more work to do a full-fledged, performant, robust and idiomatic for "both sides" python / Mathematica link, than it is to implement what ExternalEvaluate currently offers. In some sense, ExternalEvaluate is the closest thing one can imagine, to extracting a common pattern, but, as you have eloquently argued on community, it is hardly enough for the most interesting cases. – Leonid Shifrin Feb 25 '18 at 21:41
• @Szabolcs I'm not sure. I think there's enough overlap between these types of interfaces to give a reasonable overview of a few tactics / design patterns. I'm planning on writing an answer comparing a few of the ways I've written links to APIs, full programming languages, and DSLs but probably won't have enough time to write a good one until mid-week. – b3m2a1 Feb 26 '18 at 2:54
• @LeonidShifrin your first comment is exactly why I think this type of question is useful. There are many factors that go into how you design an interface, but in my experience there are often commonalities between approaches and having some sense of how the type of interface you're trying to design affects the way you design it is important and not necessarily obvious a priori. – b3m2a1 Feb 26 '18 at 2:56

Here's a repost of something I wrote elsewhere. It's still pretty partial, but lays out the basics of how I think about this.

# Types of External Interfaces

There are many possible classifications for externals interfaces, but the major two types I've run into are:

• Restricted interfaces
by this I mean interfaces where there is a proscribed set of functionality the interface should implement. Examples of this include APIs, CLI tools, and some packages.

• Unrestricted interfaces
by this I mean interfaces where there is a highly-flexible set of functionality that we want to implement. The major example of this is programming languages.

I'll discuss the restricted case first as it's much easier to handle

## Restricted Interfaces

### Git

When linking to Git there's a very small set of commands you really need to include. You can pretty much just look at the git book and determine what you need to include.

Once you've done that, all you really need to do is write a general-purpose wrapper function for running Git and then register a bunch of different functions that do minor work on top of that.

I put this into a subpackage of my main application . You can go there for all the details. In essence, though, I just wrote a bunch of little functions to handle different parts of the Git process. For example, here's how I registered the GitAdd function:

GitRegisterFunction[GitAdd, "add",
{
"DryRun"->"dry-run",
"Verbose"->"verbose",
"Force"->"force",
"Interactive"->"interactive",
"Patch"->"patch",
"Edit"->"edit",
"Update"->"update",
"NoIgnoreRemoval"->"no-ignore-removal",
"IgnoreRemoval"->"ignore-removal",
"Refresh"->"refresh",
"IgnoreErrors"->"ignore-errors",
"IgnoreMissing"->"ignore-missing",
"NoWarnEmbeddedRepo"->"no-warn-embedded-repo",
"ChangeModee"->"chmod"
}
]


This just defines the function GitAdd , the git command to call "add" , and the way to map Mathematica options to Git options.

Once I had all the functions I wanted I cooked them into a single Association to act as a router from a name to a method:

$GitActions= <| "Create"-> GitCreate, "Init"-> GitInit, "Clone"-> GitClone, "AddGitIgnore"-> GitAddGitIgnore, "AddGitExclude"-> GitAddGitExclude, "Add"-> GitAdd, "Move"-> GitMove, ..., "Archive"-> GitArchive, "SVN"-> GitSVN, "Bundle"-> GitBundle, "Daemon"-> GitDaemon, "Help"-> GitHelp, "HelpSynopsis"-> GitHelpSynopsis, "HelpDescription"-> GitHelpDescription, "HelpOptions"-> GitHelpOptions, "HelpFlags"-> GitHelpFlags, "HelpFlagMap"-> GitHelpFlagMap |>;  And I define a single function that provides the true interface to Git. I called that one Git . Then you can define it (with unnecessary parts excised) like: Git[ command_?(KeyMemberQ[$gitactions,ToLowerCase@#]&),
args___
]:=
With[{cmd=\$gitactions[ToLowerCase[command]]},
With[{r=cmd[args]},
]
];


And even better you can add autocompletions and things to make it easy to know what's there to use. E.g.:

<<BToolsExternal


And then can do things like:

Git["HelpDescription", "rm"]

"Remove files from the index, or from the working tree and the index.
git rm will not remove a file from just your working directory. (There
is no option to remove a file only from the working tree and yet keep
it in the index; use //bin//rm if you want to do that.) The files being
removed have to be identical to the tip of the branch, and no updates
to their contents can be staged in the index, though that default
behavior can be overridden with the --f option. When ----cached is given,
the staged content has to match either the tip of the branch or the
file on disk, allowing the file to be removed from just the index."


### GitHub

In that package I do a similar thing for GitHub's API, except with the important exception that the default operation for each registered function isn't to actually call the API but rather to build the HTTPRequest that the function will actually use.

## Unrestricted Interfaces

### Python

I developed a package for linking to python that I called PyTools .

To marshal Mathematica code down to a python representation I built out a symbolic Python package . This type of symbolic conversion is a powerful way to build an interface. In it Mathematica constructs are reduced to an intermediate symbolic representation which Mathematica can still easily manipulate and then further processing directions are defined on this symbolic form.

As an example, we'll see how I did it for this package. First load the package:

<<PyToolsSymbolic


Then you can take a Mathematica expression and convert it into a symbolic representation of a python expression. In general this is set up so your write python-like code in Mathematica:

ToSymbolicPython[
Import["PIL"];
img=PIL.Image[];
img.show[]
]

(*Out:*)

PyColumn[{PyImport["PIL"],PyAssign[PySymbol["img"],PyDot[PySymbol["PIL"],PySymbol[Image][]]],PyDot[PySymbol["img"],PySymbol["show"][]]}]


And we can see that this has built out a rather complicates structure to represent this simple program. We'll take it bit-by-bit:

ToSymbolicPython@Import["PIL"]

(*Out:*)

PyImport["PIL"]


This simply maps to a symbolic structure called PyImport . If we convert that to a python string:

PyImport["PIL"]//ToPython

(*Out:*)

"import PIL"


It just registers an import statement. In fact this uses Sow to make sure the import occurs at the header of the file. This is just generally good practice.

Moving onto the next piece

ToSymbolicPython[img=PIL.Image[]]

(*Out:*)

PyAssign[PySymbol["img"],PyDot[PySymbol["PIL"],PySymbol[Image][]]]


This builds out a symbolic representation of this syntax. It's inspired by the low-level representation Mathematica uses:

FullForm@Hold[img=PIL.Image[]]

(*Out:*)

Hold[Set[img,Dot[PIL,Image[]]]]


We have a syntactic wrapper for assignment ( Set ) for the . accessor in python, and a conversion of Mathematica Symbol constructs into PySymbol constructs which have less ambiguity in conversion to a string.

Finally, taking this all together, we have a PyColumn expression which is just a mimic of Mathematica's Column function which arranges pieces line-by-line after each other. And this gives a nice way to go from Mathematica-level syntax to python code:

ToSymbolicPython[
Import["PIL"];
img=PIL.Image[];
img.show[]
]//ToPython

(*Out:*)

"import PIL\nimg = PIL.Image()\nimg.show()\n"


All it took was a long-symbolic detour.

As a final note, the power of this approach is in its flexibility. For instance, if we want to register new type conversions, we need only register patterns to get from a Mathematica construct to a symbolic python one. I did this for a large set of constructs, allowing one to automatically generate code for things like:

ToSymbolicPython[
myFunc[file]:=
With[{x=Open[file], y=Open[file2, "w+"]},
Do[y.write[line], {line, x}]
]
]//ToPython//StringTrim

(*Out:*)

"def myFunc(file):\n\twith open(file) as x:\n\t\twith open(file2, 'w+') as y:\n\t\t\tfor line in x:\n\t\t\t\ty.write(line)\n\t\t\t\tNone"


### SymbolicC

SymbolicC is a package built into Mathematica that works in a similar way. I'll let its documentation speak for it.

The basic idea is to use the MathLink system to pass packets around which encode the evaluation data desired. This is incredibly flexible but requires a lot of work to do right.