29
$\begingroup$

Background

I am currently consulting with a client that is using Mathematica to prototype a number of proprietary algorithms. Many man-hours of work have gone into the development of these algorithms and there is an expectation that this work will be reused as-is.

I have been brought in to help design and develop my client's products with a focus on getting them ready for a market beta.

I am a veteran full-stack commercial software consultant - my software design and implementation practices are somewhat mature; I follow the SDLC, use waterfall, agile, scrum, lean etc and practice XP, TDD, BDD and DDD as appropriate. I work with many different tools and languages - C#, JavaScript, TypeScript, PowerShell, Bash, SQL etc.

The above information is relevant only to set the scene; my client has very little experience in commercial software development, as their focus is more on academic research. As you may imagine I face quite a bit of push back when discussing or proposing solutions to problems that I've faced many times previous, so I'm looking for some solid information that I can use to inform our decisions.

We are currently considering a few options for the software product; either outsourcing development to Wolfram or developing in-house with .Net (either importing the Wolfram code into .Net, or rewriting the algorithms in .Net). .Net is a preference because my client has other applications already written using it.

State of the Code

I am by no means an expert in Mathematica, but from what I can see the MMA prototype code is monolithic - essentially a few big blocks of text in a single notebook. I find it quite difficult to read, but in all honesty it's probably going to be dealt with as a "black box" by a product developer anyway. All they really need to know is the input and the output requirements for the functions / algorithms (unless it ends up being rewritten in another language).

The algorithms will form a small (but important) part of a much larger piece of commercial software.

It is my understanding that Mathematica is a front-end for the Wolfram language that is best suited to presentations and prototypes. If this is an incorrect assumption, please put me right!

I'm also facing a number of challenges getting the Mathematica notebook format working in any useful way with source control (git). I've looked up many solutions, but none are really suitable for day-to-day development, branching and merging.

As for integrations, I have looked at .Net / Link, but have not yet tested it to see if its suitable. Any comments on experiences around this would also be welcome.

Question

Is Mathematica suitable as a platform to develop a fully-fledged commercial software product or is there another Wolfram product that would be better suited to this?

The full product is part of a cloud-based enterprise solution and will be a cross-platform mobile app and / or SaaS web application that heavily integrates with an in-situ RESTful API (data lookups, data storage, federated identity / security etc). It may also potentially require integration with mobile device hardware, depending on how the requirements pan out.

I have briefly looked at using .wl files and Workbench but am yet to convince the consultant doing the work that this is the way to go.

Thank you in advance for any replies. I appreciate this is a long post but hopefully you experts can expedite any decisions that need to be made here!

Improve this question
$\endgroup$
6
  • $\begingroup$ Comments are not for extended discussion, let's talk in chat. $\endgroup$
    – Kuba ♦
    Sep 5, 2019 at 6:56
  • 10
    $\begingroup$ I mean I’d never try to release a Mathematica based product because it’s too fragile, but reimplementing core necessary features will be difficult and time consuming. At the end of the day if you go the Mathematica route, yes, write .wl files. That’s the only way to do real development at scale in Mathematica. But I’d say get the code out of Mathematica as soon as is feasible once you have the prototype and design down. $\endgroup$
    – b3m2a1
    Sep 7, 2019 at 21:18
  • 3
    $\begingroup$ Isn’t the point of the wolfram engine for exactly this kind if case? wolfram.com/engine $\endgroup$
    – DrMrstheMonarch
    Sep 9, 2019 at 16:11
  • $\begingroup$ I asked a related question: mathematica.stackexchange.com/q/11258/973 $\endgroup$
    – Fred Daniel Kline
    Sep 11, 2019 at 1:50
  • $\begingroup$ I know you've already mentioned this, but I think Wolfram Workbench is a great tool, and sounds like it'll work well for what you're trying to do. A good in-between between Mathematica GUI and switching to .NET. What is your trouble in convincing them? $\endgroup$
    – Max Coplan
    Sep 13, 2019 at 1:33

2 Answers 2

Reset to default
19
+100
$\begingroup$

Preface

I feel inclined to answer because I live in both of the worlds you are describing. For one, I work in science where it is more than typical to write software that has sophisticated algorithms under the hood, but cannot be compared to modern apps. This software is rarely "market-ready" in terms of user experience, let alone the principles like web-first. Due to my work on Jetbrains plugins, but also by working on, e.g. JabRef, I know how hard it is to write software that is used by many people.

Please take the following for what it is: An opinion from someone who loves Mathematica and used the system for over 15 years but doesn't think the Wolfram Language is the best tool for everything.

Answers to your statements

I am by no means an expert in Mathematica, but from what I can see the MMA prototype code is monolithic [..]

That is not true. Finished Mathematica applications or libraries should be developed as so-called packages which can contain many modular files, private functions and things that are exposed to the user. Like in other languages too, these packages are then loaded in your session and you can use the functionality. Same as import in Java or Python.

It is my understanding that Mathematica is a front-end for the Wolfram language that is best suited to presentations and prototypes. If this is an incorrect assumption, please put me right!

Yes, Mathematica consists of the front end that stores computations as notebooks and has all kinds of nice features for dealing with, e.g. graphics or laying out graphs. However, the computations are done in the kernel which is nowadays called Wolfram Engine and which is connected to the front end. The front end, however, is essential if you want to interact with graphics, images, sound, etc. or want to make dynamic plots or UI.

I'm also facing a number of challenges getting the Mathematica notebook format working in any useful way with source control (git).

Just don't. Working with plain-text packages solves these problems. Look for instance at the IGraphM package of Szabolcs which is an excellent show-case of a modular Mathematica package which is actively developed for a long time.

As for integrations, I have looked at .Net / Link, but have not yet tested it to see if it's suitable. Any comments on experiences around this would also be welcome

The links to Java and C work excellent, and I'm sure .Net is only a wrapper around the C link (called MathLink or nowadays WSTP) and works equally well. However, the crucial part is that when you use Mathematica functionality, you will always need a Wolfram Engine in the back that evaluates your requests. So your users will either need a local license or if you are running a web-service, you will need some of the cloud-stuff or a webMathematica. You should not forget that this means you have running costs. To say it plainly: You cannot deploy a Wolfram package which integrates into your app without needing a Wolfram Engine.

My advice

If the Wolfram code doesn't need to be extended in the future and it doesn't use advanced algorithms available in Mathematica, I would think hard if it isn't better to re-implement it. This has the following advantages:

  • You don't need a WolframEngine in the back which saves money and prevents you from errors/hangs/slowness which are not under your control
  • You can debug and maintain code in a language you are used to and you won't run into surprises from a "black box" Wolfram code
  • You can ship your app to whichever system you like. I might be wrong, but I haven't heard that the WolframEngine runs on Android phones

Therefore, in a first step you should carefully assess the Mathematica code you are dealing with. The big question is, does it use advanced functionality that will be hard to re-implement? You should look out for the usage of

  • complex plots or visualizations. For instance, ContourPlot, RegionPlot, but others as well contain sophisticated algorithms to create what is finally displayed as graphics or 3D graphics.
  • symbolic calculations like Integrate, solving differential equations, optimization (Minimize), or working with statistical distributions. But also stuff like simplifying large symbolic expressions or the usage of special functions
  • advanced numerical algorithms used for optimization, statistical testing, integration, image processing, etc.

Update: What you could do, hopefully without breaking any NDA, is to share the list of system functions your monolithic notebook is using. This would help to point out potential problem-functions. You can use something like this:

file = "/path/to/your/notebook.nb";
Cases[
  NotebookImport[file, "Input" -> "HeldExpression"],
  s_Symbol /; Context[s] =!= "Global`" :> HoldForm[s],
  Infinity,
  Heads -> True
  ] // Union

This would give a first glimpse of what you are dealing with.

To give you a specific use-case: If all your Wolfram code is doing is handling data and calculating, e.g. statistical things like mean, median, variance, etc., I'm sure you can easily re-implement this. However, if you are relying on a ContourPlot3D, you need algorithms to find the contour of a possibly complicated function, create the polygons and display them as nicely as Mathematica does. That's more work to re-implement or find appropriate libraries.

The next question is, how likely is it that your client will make changes on the Wolfram code or incorporate new features? If the functionalities are final and you can re-implement all algorithms, then I would opt for a re-implementation. However, be aware that only one single line of Mathematica code that needs to be updated can bring you in big trouble on the .Net side because things like TextRecognize will be hard to re-implement in the same way they work in Mathematica.

The full product is part of a cloud-based enterprise solution and will be a cross-platform mobile app and / or SaaS web application that heavily integrates with an in-situ RESTful API (data lookups, data storage, federated identity / security etc). It may also potentially require integration with mobile device hardware, depending on how the requirements pan out.

I might be wrong here, but I'm not aware that the Wolfram Engine runs on mobile devices. Therefore, you will very likely need to call it on a server running the Wolfram Engine. This is a bit out of my comfort zone but this should mean your server needs to provide computational Wolfram power to many requests with low latency, right? I would think really hard if I need to go down this path.

I guess that's most of the things I can say without looking at the actual code you are dealing with. If you have further questions, let me know.

Improve this answer
$\endgroup$
7
  • 2
    $\begingroup$ This has been a wonderful and well considered read. You are essentially asking the same questions that I am asking my client and are outlining the same conclusions that I'm coming to. What is particularly interesting is your second set of bullet points; I do not know MMA enough to make these calls - I cannot find any of the functions you mention in the code, but I don't know if the other features are being used. $\endgroup$
    – Spikeh
    Sep 11, 2019 at 5:51
  • 2
    $\begingroup$ I have updated my answer and provided a small snippet of code in the middle that lets you extract all used functions. If you can share this list in your question, it probably becomes a bit clearer what you are dealing with. $\endgroup$
    – halirutan ♦
    Sep 11, 2019 at 6:39
  • $\begingroup$ You mention "The front end, however, is essential if you want to interact with graphics, images, sound, etc." For manipulatable objects, this may be true. But there are certainly other ways of using those objects without Mathematica. See WolframLanguageForJupyter $\endgroup$
    – Max Coplan
    Sep 13, 2019 at 1:39
  • 4
    $\begingroup$ @MaxCoplan That is the reason why I wrote: "interact". In Jupyter, all polygons, lines, etc. are rasterized into an image. This is a vast difference to Mathematica, where you can rotate 3d graphics or edit graphics primitives in the front end. Basic example: Make a BarChart in Jupyter and try if you can see the tooltips. Getting a simple image is different to having a dynamic vector-graphics representation. $\endgroup$
    – halirutan ♦
    Sep 13, 2019 at 1:57
  • 1
    $\begingroup$ @MaxCoplan There really is no apology required. It's just sad that we simply don't have a way to export, e.g. a Graphics3D objects with all its details in a general format that would allow rendering the object with OpenGL (or WebGL). Jan's latest notebook embedder does this implicitly, but I haven't had a close look at it. $\endgroup$
    – halirutan ♦
    Sep 14, 2019 at 18:17
4
$\begingroup$

I agree with the comments above, that is, using packages written as pure text .wl files. Integrating Wolfram Language code as part of an enterprise solution is not an easy task. I recommend contacting Wolfram Technical Services. They have significant experience in development and integration at this scale.

http://www.wolfram.com/technical-services/?source=frontpage-features

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ After I posted this I actually engaged with Wolfram's technical services in a face to face meeting. They confirmed much of the above - and were surprisingly unbiased in their opinions and recommendations. $\endgroup$
    – Spikeh
    Sep 11, 2019 at 10:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.