# What's the deal with next generation Wolfram compiler?

Sources:

Does anyone here has come across this new next-generation Wolfram compiler thing? Will it be able to compile the entire Wolfram language? As far as I understand Wolfram language is an interpreted language, so are they turning it into a compiled language. Will we be able to create standalone executables like in C and C++?

Will this compiled Wolfram language support the compilation of its symbolic capabilities?

• I'm pretty sure it is going to be something like F# Compiler Services. This would be a great development, since it would allow for things like trans-compilation from WL to JS, for instance. – Shredderroy May 15 '18 at 17:08
• "When" is also an important question, I understand it is a major project yet it is advertised at least since 2014. – Kuba May 15 '18 at 21:05
• @Kuba With something as fundamental as this, I'd rather wait than see something half-baked released (and possibly left in that state). – Szabolcs May 16 '18 at 12:32
• @Szabolcs sure, I don't claim it takes too long, only that I don't even follow this since 2016 because as an inexperienced user I can't tell if there was any progress. And if it is almost done or 5 more years is something to expect. – Kuba May 16 '18 at 12:38

First, no one outside of Wolfram could use the compiler so far.

From what I have seen, it will be very much comparable to what the Julia language does. If you watched the videos closely, you'll notice that the new compiler too, will contain several steps that gradually transform a Mathematica expression into its compiled form by analyzing types and pressing variables and functions into a type system.

That being said, I believe it won't be possible to compile everything. The strength of Mathematica is that everything is an expression and it is evaluated until the final, normal expression is reached. If I should make an educated guess, then something simple like this

func[i_Integer] := Switch[i, 1, 100, 2, "Hello", 3, Missing[], _, \$Failed]


will not be compilable, because the return type of func cannot be pressed into some type category. This, however, implies that many high-level functions cannot be compiled down. In compiled code, the types of functions or expressions need to be fixed.

Here are some points at which to my understanding the new compiler will be awesome:

1. Automatic deduction of types. This means, that unlike now, you don't have to specify types for all of your variables. Right now, if This is similar to what happens in Julia or Kotlin. Some types can directly be deduced by inspecting the code.

2. The compiler will be able to create several instances of the same function which use different types. So a simple function like f[a_,b_]:=a+b will have several instances that work with integers or floats depending on the input values.

3. We finally have recursive calls. This is a big deal because at the moment, we cannot do this with Compile

4. I hope for easier support of including compiled functions into other compiled functions especially when it comes to parallelization.

5. I'm keen to see the optimizations possible at the LLVM stage. In Tom's latest video, there is a slide where he showed how you can turn on LLVM optimizations of the code that the compiler created. That looks extremely interesting

6. Since the new compiler will have a LLVM stage, we are in general no longer dependent what the final target is. So it is surely possible that after some time, we could compile Mathematica to JS.

All that I mentioned is purely deducted from the videos that were published. I have not insider information of any kind. But I'm one of the people who is desperately waiting for the compiler for several years. If it would have mattered, I would have sent TWJ coffee, mate-tea, or pure caffeine just to make him work faster on this. I really want to test this.

I should elaborate a bit on this point as I feel it is too vague. Currently, we also have type inference. Consider this simple example

Compile[{}, Module[{x}, x = 1; x + 1.2]]


The type of x is deduced from its usage and the result is as expected a real in the end. Common practice is to give x some fake initial value directly inside the Module list to ensure that it is of the correct type. Inside Compile, this is the only way to specify the type if it cannot be deduced automatically.

A slightly more complicated example is when you want to say, x is a list that contains elements where each element is a list of reals. To make this happen, you have to show this to the compiler because the following doesn't work

Compile[{},
Module[{x = {}},
AppendTo[x, {1.0}]
]
]


You could say x = {{1.0}}, but then your list already contains an element. The trick to be used is to show the Compiler a valid list and instantly remove the first element again

Compile[{},
Module[{x = Most[{{1.0}}]},
AppendTo[x, {2.0}]
]
]


That is not so nice. So the hope is, that the new compiler either can do some advanced inference and is able to deduce the type of x from the call

AppendTo[x, {2.0}]


itself, or that we can simply say x is a list of real lists. Related to this is the usage of InternalBag` inside a compiled function.

• Ah, so it will move compiling from art to science domain? I said art because it is really tough to keep track of how to use Compile unless you do this on daily basis. – Kuba May 16 '18 at 13:58
• I have a question about 'it won't be possible to compile everything'. Can't they create/include 'expression' class which can be used when type deduction fails? The only requirement would be to have a valid 'main' defined. – Kuba Jan 16 '19 at 9:29