Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I am an undergraduate student, starting to use Mathematica to model simple physical chemistry problems, as a prelude to a summer internship in a computational/theoretical chemistry group. The solution to a problem I was given (and walked through) now needs to be incorporated in the group's program, which contains >17 Kloc. Before seeing this big program, my experience with Mathematica gave me the impression that it was a very sophisticated graphical calculator.

Mathematica is fast, extremely versatile, but its code cannot be compiled and made executable (for what I know). I know one can use the CDF player, but that means the program is not really standalone. Considering my example, I am wondering if there is a point at which one would be better off using another programming language than Mathematica to do the computation. Are there disadvantages or obstacles present when Mathematica programs grow bigger than a certain size?

share|improve this question
10  
This is just personal opinion but I think Mathematica shines when it's used for interactive work, and not for developing end-user applications. –  Szabolcs Feb 4 '12 at 18:43
3  
I think your question and it's title are not in very good agreement. As for the title: yes, of course it can be regarded as that, and even more important, it can and has been used as such. As for the question in the body: Decisions on which language to use for a given project depend a lot on who will be writing the program and by whom and how it is going to used. Technical reasoning is usually by far not the most important thing when such decisions have to be made (think about acceptance,licensing,prior knowledge and all that). Probably you want to elaborate what your situation is... –  Albert Retey Feb 4 '12 at 19:52
    
C was just an example. I updated the question. –  CHM Feb 4 '12 at 23:43
3  
Are you saying that the computational chemistry group has 17Kloc of code in one notebook used and changed by various members? It's always amazing how people manage to get their things done the hard way :-). If that's the case and members of the group are comfortable and happy with Mathematica I don't think there is much sense in changing the language. But probably you should try to propose some of the suggestions Leonid and I have mentioned (split in modules, use version control,...) -- and be open to learn that there might even be good reasons for what the current situation is... –  Albert Retey Feb 6 '12 at 0:17
    
@AlbertRetey Since I have been working on this project for just over a month, I have no clue as to how the team manages its software. I am sure there are good reasons for it to be the way it is. I was only wondering if there were any challenges associated with such big programs. I am glad to know what things to learn about, because I surely won't make any suggestion to the group before actually understanding 1) the code they wrote and 2) what benefits such and such change would bring. It is a learning experience that I want to get the most out of, both in chemistry and programming. Thanks –  CHM Feb 6 '12 at 2:12

4 Answers 4

up vote 39 down vote accepted

I think that Mathematica is a great prototyping environment, and has a bright future as a system for both prototyping and implementation of complete components of other systems, from back-ends to front-ends. In my opinion, we are now witnessing the process of it being transferred from pure scientific tool to a general software engineering tool / language.

So, I think that moving to another language can be often done pretty late in the development cycle (disclaimer: I have not personally built large systems involving Mathematica as a part - although I worked on large systems written in Java before - so what I write here is mostly an educated guess based on my separate experiences in Mathematica and other languages). The great benefit of Mathematica is that it is a very high-level development environment which can serve as a gluing medium for development of hybrid systems, where different parts are written in different languages. For example, I found it a great testing / development medium for Java applications. This is generally not yet quite apparent since we still lack some tools to boost productivity and overcome cross-lnaguage barriers. But I am more than positive that such tools are going to emerge pretty soon. When you develop the system, what matters is how flexible is your architecture, how testable are your modules, and how fast are the development iterations. A high-level environment like Mathematica is a great win for all these.

That said, I would not currently use Mathematica as a central run-time of the application, simply because the kernel crashes every now and then. I would make that another runtime (e.g. Java), which calls Mathematica and handles possible errors, exceptions, crashes and the like (actually, WebMathematica is just that - Mathematica managed by the Java runtime and bundled as a web application for some Java container like Apache Tomcat). Mathematica can however serve as both an excellent back-end and an excellent prototyping environment, so once again, my feeling is that one can benefit a lot from developing even large industrial systems in or with Mathematica. There are actually companies which do just that, and are quite successful.

As to when to use C etc - my advice is: as late as you can. Many problems for which Mathematica is perceived as slow can be solved quite efficiently with the knowledge of how to write efficient Mathematica code. May be even more importantly, it is rare that you know the exact method you will use for a given problem, all in advance. Once you switch to C, you will have to deal with lots of low - level details, which will increase development time and chances for errors, plus they will distract you from the essence of the problem you are solving. Even if you switch to C at the end, Mathematica can save you a lot of time in prototyping your solution, and minimize the amount of low-level work you have to do.

Scaling to large code bases:

This is a problem in pretty much every language. There are probably many factors which determine how well a given language scales. Part of this is also probably not just about language itself, but about existing development tools. For example, Java scales reasonably well, but no one in their right mind would use it for large projects without smart IDE-s. So, I'd set out a few important factors (a list is incomplete, of course):

  • Type system. Strongly typed languages can use the compiler to help find errors, and this will be particularly powerful for those with type inference (ML family languages for example).
  • Means for composition. These include classes / interfaces / inheritance for OO, and higher-order functions / closures / possibly macros for FP. I am biased towards FP here.
  • Means for information hiding, and separation between interface and implementation. This is extremely important, and this is where OO shines, IMO. You can get it in FP, but have to be more disciplined.
  • Package / module system, and namespaces - this is a very important tool for large-scale encapsulation / information-hiding
  • Development tools (IDE-s, debuggers, profilers) - can make a huge difference.
  • Standards of coding and code exchange. When they exist, it makes for much easier code reuse, assuming that you don't write everything yourself.

There are probably other important factors I missed. The question is how does Mathematica fare regarding these factors. I'd say that potentially, Mathematica can fare quite well. I think right now it suffers the most from a lack of certain development tools (a really good / useful debugger, for one) and coding / code exchange standards. Also, the programming practices which allow to scale to larger systems, while certainly possible in Mathematica, are not developed / not in widespread use yet. For example, closures and higher-order functions are very useful for that, but it's not something every second Mathematica programmer is using. Also, while Mathematica allows to write macros (functions which manipulate code), its rather complex evaluation control mechanisms make them hard to write. And macros are the extremley powerful scaling tool - in LISP they allow for easy creation of DSL-s because essentially they extend the compiler in the direction you want. Another problematic thing is that Mathematica is often too general, and this generality gets in the way in forms of evaluation and performance surprises. Some intermediate language layer would be a big help here.

To summarize, my opinion is this; Mathematica can be used for large projects, even at present (it actually is used for at least two huge ones: it is written largely in itself, and WolframAlpha is another example. From my personal experience, a few of my projects were several thousand lines long), but your code won't scale automatically for you, and you need to be a pretty good Mathematica programmer to be able to manage the complexity of large projects. In this regard, many modern languages provide more automatic tools for scaling the code base, and more tecniques are well-known and in widespread use. I also think, that the situation with Mathematica will improve in the future, we will have better development tools, more programming practices will be shared, etc. So, yes, you definitely can use it for large projects, but right now it won't be as easy as say in Java, Python, or some other well-known languages. Much of it is not at all inherent in Mathematica per se, but reflects its young age as a general-purpose programming language used for larger projects outside academia. My two cents.

share|improve this answer
4  
Ditto. I've been using mma to design C code since 1981 when it was called SMP. –  Reb.Cabin Feb 4 '12 at 21:25
    
I agree as well. I like being able to work in as high a level of abstraction as possible (or to use an analogy, I'd very much prefer to write in C than in assembly language), and Mathematica provides a pretty high level of abstraction. When I need to switch to another language, only then do I worry about lowering, and even then, I only have to worry about components instead of a monolith... –  J. M. Feb 4 '12 at 23:53
    
Thanks for the great answer, it was what I hoped for. My knowledge of computer programming is limited, and your answer makes it possible for me to go read about concepts I don't know much about, or have never heard of, such as strongly typed languages, FP vs OO, etc. Thank you. –  CHM Feb 5 '12 at 1:19
1  
@CHM: On that note: may I recommend picking up Steve McConnell's Code Complete, among other books. It's a nicely written set of guideline you might want to consider adopting as you go on and write increasingly intricate routines. –  J. M. Feb 5 '12 at 10:35
    
@CHM Glad I could help. Thanks for the accept. A small extra piece of advice: if you want to learn to scale your code, pick some language which is currently good at it and has lots of learning resources on this matter (this includes open source projects) available (Python and Ruby come to mind for more dynamic stuff, and ML/Ocaml/F# or Scala if you want stronger typing), and do a moderately large project in them, or get involved in one. The knowledge you gain there, can be easily transferred to Mathematica, and this may be more effective than learning this within Mathematica only. –  Leonid Shifrin Feb 5 '12 at 14:16

I completely agree with Leonid but for one nuance:

If, like me, you are some sort of quantitative professional, but not a programmer or computer scientist, then Mathematica offers power and flexibility for building your own applications much more efficiently than many other options. True, it doesn't allow you to build standalone applications: the users will always need Mathematica or Player Pro, or you will need to abide by the restrictions of CDF and the Demonstrations format. But what Mathematica does do is enable people who are not professional programmers, who don't want to learn Java or C or some other lower-level language, to build useful end-user applications.

The key advance was the dynamic interactivity functionality introduced in version 6, including Manipulate for simple things and DynamicModule for more complicated things. Building dynamic graphical interfaces for users, with sliders and buttons and so on, is hard. It is normally the domain of professional programmers. But Mathematica puts that power in the hands of people who are not professional programmers and have enough on their plates with their real jobs. It makes it easy by comparison to, say, building an iPhone app - and that is a platform with a lot of built-in functionality for developers.

In my own work, I was able to mock up a not-quite-working interface for a custom charting application in four hours, with enough functionality to convince the top management at my company to consider it as an option for this purpose. I could never have done that with Python or Java or C or Fortran or any of the other languages commonly used in technical fields like economics/finance or engineering. (Mike Honeychurch, who is active on this site, has some nice examples of end-user interfaces in Mathematica on his blog.)

Regarding large projects, the important thing is to modularize. You can use Workbench to manage large projects containing multiple linked packages. I recommend Wolfram's white paper Building Large Software Systems, available via the download link at this page, after giving your email addreess. It sounds like the notebook you are looking at could be split up into a package defining the custom functions, and then a notebook that actually reads in the data, manipulates and visualises it. The main package in the graphing system I mentioned above runs to 2600 lines and I don't find this unwieldy in Workbench.

share|improve this answer
    
Very good points - +1. –  Leonid Shifrin Feb 5 '12 at 10:42
    
Totally agree. I'm not a "programmer" in the sense that I don't pass more than 1 hour per day typing code. But in one our of Mathematica coding I can go much further in my developments (and most specially, thoughts) than I would be able with another language. I'm currently also thinking on a bigger development and the question is also in my mind: Mathematica or other thing? In my field, no one knows what is Mathematica, and this, together with the fact that there's no standalone (end users need to pay licenses), are the main drawbacks (who authorizes the payment for an unknown software...). –  P. Fonseca Feb 5 '12 at 11:04
2  
@P.Fonseca although I don't think that technically there is any reason to exclude Mathematica for your large project, I would not underestimate the social and economic aspects when making such a decision. If your users or co-developers will not accept your decision you'll have a hard time. Unfortunately the decision on which language to take is often dominated by technical and -- even worse -- ideological -- reasoning. In retrospect I found the social and economical aspects much more relevant for project success in almost all cases I was involved. –  Albert Retey Feb 5 '12 at 11:24
1  
The fact that the player pro doesn't represent a lot of the development price also means that most likely WR is not making a lot of money on it. And if so, I don't understand why not free. The difference in being free or having a small price is enormous for the decision makers, since the human mind doesn't compare both on the most reasonable way (and I'm also human...) –  P. Fonseca Feb 5 '12 at 11:47
1  
@P.Fonseca: I don't feel I can make a well founded suggestion for your project, but I think you got the idea: as for the point you mention Mathematica seem a good choice. Consider the other points I mentioned and everything else that could be relevant to your situation. Just make sure it is not just personal preference (which most often equals laziness to learn something new) that affects your decision. For such a project you'll have to justify your decision and it's good to be prepared for that. Knowing and considering alternatives helps a lot in that respect... –  Albert Retey Feb 5 '12 at 15:38

I have seen your edit, and also Leonids additions to his answer which I don't have to add too many things of relevance for your situation: single programmer, single user(?), large code base. Wow, 17K lines of code -- you can implement a lot of functionality with that in Mathematica...

Basically it can be read inbetween the lines of Leonids answer, but I would want to emphazise that you really should learn about some standard techniques of code organization -- which are to a great extent independent of the language/system you use. As a first step you should look into those features that Mathematica offers to better organize your code. For that amount of code you definitely should think about splitting your code into independent modules/packages and put those in separate files. Learn about how Mathematica supports namespaces (contexts, BeginPackage EndPackage and friends). I would suggest that you have a look for workbench although it is also possible to have things decently organized without it just as well. I would also suggest you start to use a version control system for such a code base (these usually like the plain text package files better than notebook files). Of course I assume you have already organized your "routines" as functions with localized variables, if not do that as a zeroth step. Splitting your code into modules will help a lot especially if you decide that for whatever reason parts of your code should be reimplemented in another language.

As my comment mentions I think the decision to change from Mathematica to another language in most cases is not so much about technical necessity really. For what I know from your situation I don't see any relevant reason to move to another language other than probably if you need maximal numeric performance in some routines or have collaborators that wouldn't accept Mathematica code. In other situation that decision usually needs to take things into account that I think are probably not that relevant to you, but in general should be considered, like:

  • Who will be co-developers, will they accept Mathematica and know it well enough to be productive? (that probably is the most relevant point: my experience is that you will always fail if you force programmers to use something they don't like: it's just too easy to shift responsibility of failure to the system that was enforced to use. Even worse when they actively look for such situations).
  • Who and how many users will make use of it? Often there is a breakeven for costs of additional implementation effort vs. costs of per user license fees. For only a few user usually no additional effort is justified, but if you have several thousend users licensing fees even for the Player Pro become an important factor. It might also become a problem that with a system like Mathematica under the hood you don't have always full control to fix bugs or implement features exactly as users might specify (e.g. setting focus or reaction to keyboard input in guis).
  • How will the program be run: will it be run once a year or e.g. regularly every hour? Will it need changes between runs or not? Which kind of computers will it need to run on (think about things like supercomputers, mainframes, GPGPU, cloud computing services, tablet pcs, smartphones)? Are there any regulatorities in the field of application which you might not be able to conform to when using Mathematica? Again there is a tradoff between additional efforts to reimplement functionality in lower level languages vs. potential speedup or demands on resources.

I rethought my own decisions of the past years and found some more relevant points, so I'd like to add these:

  • Who will be responsible for installation and support of your program? Can you convince them to support your decision? Keep in mind that Mathematica is often somewhat exotic for IT personell to work with. If they are not supportive, it's just so easy for them to destroy the reputation of your program (and Mathematica as a system).
  • Avoid getting trapped into ideological debates, they are often emphazising points that are of minor relevance in practice (IMHO). Typical topics are OO vs. FP, open source vs. commercial, interpreted vs. compiled, strongly typed vs. weakly typed. All of these have their place, advantages and disadvantages. In the end it seems to be of more relevance how developers actually make use of these concepts than what was actually chosen (Bad code can be written in every language). Making use of the enthusiasm of people trying to proof their point might be a good choice in that respect :-).
share|improve this answer
    
Great answer, Albert! Would vote more than once if I could. –  Leonid Shifrin Feb 5 '12 at 10:40
    
thanks, I appreciate it. –  Albert Retey Feb 5 '12 at 11:19
    
Regarding installation - I'd strongly suggest using web interface, if at all possible (make it a web app). The web part can be done with WebMathematica, or some separate web framework (Rails, Grails, Django, whatever). Data to/from Mathematica can be sent / received via AJAX calls. You may not get the full user experience you can get in Mathematica, and it may be more work, but then there is no installation whatsoever, which is a huge plus. Support is a different matter of course. But people are much more keen on web interfaces now, so convincing may be easier. –  Leonid Shifrin Feb 5 '12 at 17:56
    
@Leonid: I agree with that for a great many use cases, but certainly not for all. I hope that one of my main points is clear: you really have to look at all the boundary conditions to the project/program at hand, and only then you can make a well founded decision on which language, technology, user interface etc. is to be used. –  Albert Retey Feb 5 '12 at 22:28
1  
Again @Leonid: And don't forget that a web application also needs to be installed and maintained -- on the server. I have a few years of experience in that field and remember that installing server applications was always a nightmare: because these would presumably only need to be installed once no one cared about making these friendly to install. But as time goes by hardware changes, there are OS upgrades and you'll be installing every now and then anyways. The situation might have improved with all the virtual servers used nowadays, but still, someone has to do it once in a while. –  Albert Retey Feb 5 '12 at 22:34

Theo Gray talks about the value of prototyping with Mathematica in his recent talk about his ebooks. The talk is on YouTube here. In this video, he explains that he didn't know whether it would be possible to do some stereo/optical interface, and, once he found out that it was possible (in Mathematica), he had the final version written in C.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.