45
$\begingroup$

I'm just curious. My friend just told me that Mathematica is mostly for symbolic calculation and not efficient for Numerical computations. He told me that's the reason most of the people don't use Mathematica for CFD and other numerical intensive code.

I've just started with Mathematica (I don't know C and Fortran). I was assuming that since Mathematica is new when compared to C and Fortran it should have included all the problems that C and Fortran might have, and since Mathematica has many inbuilt functions it should run faster than C and Fortran.

Why is this not the case?

Is there any case where Mathematica's code runs faster than C and Fortran?

$\endgroup$
  • 3
    $\begingroup$ Szabolcs in his answer demonstrates in a beautiful way that the opinion raised by your friend that Mathematica is "not good with numerical computations" is often just simply incorrect. It is correct that if you are a very good C++ programmer you may be able to implement a solution which runs faster than with Mathematica. However, even relatively large numerical tasks can be implemented successfully with Mathematica IF you make an effort to learn Mathematica properly, and do not generally use For loops and similar constructions you would use in a low level language. $\endgroup$ – Michael Weyrauch Oct 17 at 8:51
  • 3
    $\begingroup$ Normally one says that there are no bad questions, only bad answers. However, here I believe this is a rare example of the "incorrect question". The point is that one can deliberately make any code as slow as desired. $\endgroup$ – yarchik Oct 17 at 9:50
  • 1
    $\begingroup$ For simple tasks, like vector addition, its much easier to write efficient code in Mathematica. When you have a complex problem though, like some CFD calculation, it's just as hard to write good code and I personally find C/Fortran more forgiving when you are a beginner. $\endgroup$ – banone Oct 17 at 14:37
  • 1
    $\begingroup$ @GummalaNavneeth: In C, "perfect code" won't be portable; it will be optimized for the specific machine you're running on (e.g. Skylake Xeon), using SIMD intrinsics like _mm512_add_ps and _mm_shuffle_ps. Plus tuning for cache-blocking / loop-tiling for some specific L1d or L2 cache size in a matrix multiply, and so on. In Mathematica that level of detail is hidden in the implementation of array / matrix operators, and will be pretty good on any machine that Mathematica runs on. $\endgroup$ – Peter Cordes Oct 17 at 18:24
  • 3
    $\begingroup$ Your friend's old saying was probably true until version 4, when packed arrays were introduced. At that point, WRI realized, apparently, that efficient, powerful numerics were needed to go alongside their symbolics. To date, the advancement of symbolic preprocessing means the ordinary user can have a likely algorithm chosen for them that will be numerically efficient. If that fails, they can resort to the old method of consulting an expert. $\endgroup$ – Michael E2 Oct 18 at 12:57
74
$\begingroup$

High-level languages, like Mathematica, have a high overhead for executing each command/instruction. However, they also typically include commands/instructions that solve a larger and more complex task than those in low-level languages.

To take a concrete example, in C, we can add two numbers. In Mathematica, we can add two arrays directly. If we want to do the same in C, we must write an explicit loop, and implement array addition in terms of the more basic scalar addition. I wrote a small benchmark to compare a naïve C++ implementation (c[i] = a[i] + b[i]) to Mathematica's builtin. Mathematica's is 2.7 times faster. How can this be? It is because Mathematica's array addition is not implemented in a naïve way. A lot of effort was put in to create a very fast implementation that might make use of SIMD instructions and multithreading. Can you do this in C++? Of course, but it takes much more effort, more time, more expertise. In Mathematica, even a complete beginner can use array addition.

It's not as simple as "is this language is faster than that language".

  • Low-level languages give you small and simple building blocks. Using the building blocks has very low overhead. Since we must build everything from the smallest and simplest pieces, building things takes more time and effort.

  • High-level languages give you larger building blocks, each of which accomplishes a more complex task. Using the building blocks has high overhead, so if you need to put many of them together, the result will be slow. If you can phrase your problem in terms of just a few building blocks, then the high-level language has the advantage.

For example, if the solution to a task can be expressed in terms of matrix arithmetic, and the matrices are large (thus each operation takes much longer to complete than its overhead), then it is better to use the high-level language. If there is already a function in the high-level language that solves your problem, it is better to use it.

Sometimes you need to develop a custom solution for a problem, for example, implement a new CFD method. There is no existing implementation that is accessible from some high-level language. Your only choice is to implement it from the most basic building blocks: loops and arithmetic. In this case, the only good choices are low-level languages.


The benchmark

This benchmark compares a naïve C++ implementation of vector addition to Mathematica's built-in vector addition. I use my LTemplate package to save some effort in connecting the C++ program to Mathematica, but this is entirely irrelevant for the benchmark.

Needs["LTemplate`"]

SetDirectory@CreateDirectory[]

template = LClass["Adder",
   {LFun["add", {{Real, 1, "Constant"}, {Real, 1, "Constant"}}, {Real,
       1}]}
   ];

code = "
  struct Adder {
    mma::RealTensorRef add(mma::RealTensorRef a, mma::RealTensorRef b) {
        auto res = mma::makeVector<double>(a.size());
        for (mint i=0; i < res.size(); ++i)
            res[i] = a[i] + b[i];
        return res;
    }
  };
  ";
Export["Adder.h", code, "String"];

CompileTemplate[template]

LoadTemplate[template]

adder = Make[Adder]

a = RandomReal[1, 100000000];
b = RandomReal[1, 100000000];

RepeatedTiming[c1 = adder@"add"[a, b];, 10]
(* {0.4838, Null} *)

RepeatedTiming[c2 = a + b;, 10]
(* {0.1809, Null} *)

c1 == c2
(* True *)

Benchmarking environment: Mathematica 12.0.0, Ubuntu 16.04, GCC 6.5.0, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz, compilation flags are CreateLibrary's defaults (i.e. -O2) amended with -std=c++11. The timings shown here are the minimum of 10 runs (each for 10 seconds with RepeatedTiming).

$\endgroup$
  • 2
    $\begingroup$ I disagree. The dichotomy between high-level and high-performance is nowhere near as this answer suggests. I think what you actually mean is, “Dynamical languages, like Mathematica, have a high overhead”. But there are other high-level languages that are perfectly capable of pretty high performance. O'Caml is a good example. Sure – they have other tradeoffs, they are still not quite as fast as hand-optimised C, and you may think them not as convenient as Mathematica. But especially for something like “implement a new CFD method” they can make much more sense than going all the way low-level. $\endgroup$ – leftaroundabout Oct 17 at 14:26
  • $\begingroup$ It's not a fair comparison when you use Mathematica's built-in optimized function to add two lists on the one side and a naïve addition loop instead of an optimized linear algebra library. To be fair, it should either compare addition loops in both languages, or use optimized solutions in both. Not sure about C, but for C++ I'd try Eigen (with compiler optimizations, of course) as the optimized solution. $\endgroup$ – Ruslan Oct 17 at 15:20
  • 18
    $\begingroup$ @Ruslan I believe using a naive implementation is exactly the point. I have no doubt that any problem can be solved faster with C/C++ but developing the program to do it would probably take much longer. It's a question of trade offs. How much time developing a custom solution are you willing to and how fast do you need the solution to be? This is expressed in Scabolcs answer clearly in the "Sometimes you need to develop a custom solution for a problem" paragraph. $\endgroup$ – Captain Man Oct 17 at 15:31
  • 2
    $\begingroup$ Overall I mostly agree with this answer. (And I say this as someone who does know how to write low-level optimized stuff in C or C++, with SIMD intrinsics and/or asm tuned for current Intel CPUs, including things like cache-blocking.) Very high level languages like Mathematica or NumPy are great for gluing together highly-optimized building blocks without much dev time. That's good, but usually each temporary array / matrix actually gets written in memory, instead of doing multiple steps while data is hot in cache. Computational intensity (ALU per memory bandwidth) can be a problem. $\endgroup$ – Peter Cordes Oct 17 at 17:58
  • 3
    $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. I left few comments as a teaser but if authors think they alone don't represent their point well they can delete them of course. $\endgroup$ – Kuba Oct 18 at 5:59
3
$\begingroup$

Mathematica is written in C/C++ itself.

That means it cannot run faster than C/C++.

(Assuming of course the faster implementations are used for each algorithm).

$\endgroup$
  • $\begingroup$ I'm sorry since I don't know much about coding. Could you elaborate on your answer?. I'm assuming that If Mathematica is written in C/C++ it should be as fast as C/C++. Why does it run slower( I actually don't know whether Mathematica runs slower or faster compared to C/C++, as I don't know C/C++ ) for high numerical computations? $\endgroup$ – Gummala Navneeth Oct 18 at 2:50
  • 6
    $\begingroup$ @GummalaNavneeth: This answer is kind of true in a literal sense: you can recreate the performance of a Mathematica program running inside Mathematica by implementing Mathematica yourself, with all the same optimized libraries it uses. With that as a hypothetical starting point to optimize from, you can't do worse than Mathematica (if you benchmark), and might easily do better by hand-optimizing away some of the interpreter overhead. Of course it's not practical because Mathematica is closed source and incorporates many person-years of dev time which you couldn't practically duplicate. $\endgroup$ – Peter Cordes Oct 18 at 4:31
  • 4
    $\begingroup$ You might assume this is the case, but it's not. There's all sorts of magical techniques like platform-specific JIT compilation and runtime reoptimisations that can make a higher level language faster than lower level languages. Things you COULD do but that aren't practical in C++. It's not a hard time. Sometimes Java will be faster than C++. $\endgroup$ – Benny Mackney Oct 18 at 7:55
  • 1
    $\begingroup$ @BennyMackney It is that way. Assuming perfectly optimised C++ code, Mathematica can never beat that, because it is written in C++ itself. In the best case scenario, Mathematica would reach equal timing. Otherwise, you could "simply" recode Mathematica in C++... $\endgroup$ – infinitezero Oct 18 at 10:04
  • 1
    $\begingroup$ @BennyMackney actually, you don't have to code the JIT system itself. There do exist JIT compilers for C and C++, see this question at Software Engineering SE. So assuming that a normally-low-level language must be compiled as such is also not fair :D $\endgroup$ – Ruslan Oct 19 at 10:30

protected by xzczd Oct 18 at 7:30

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.