What I am most interested in is described in Oleksandr's answer. Here's something else I would also try:
There are several built-in functions that take advantage of parallelization without any special settings (and without using the Parallel tools framework). I would like to know how well these scale to a high number of cores.
Examples:
Matrix operations, such as addition and matrix multiplication typically take advantage of multiple cores. I would try how Mathematica performs on tasks such as eigenvalue computations for dense and sparse matrices for very large matrix sizes. Large matrices take a lot of storage space, so you'll also make use of the memory. For large matrices it'll be realistic to compute only the few largest eigenvalues.
I noticed that NIntegrate can also use multiple cores. I would try how much speedup can be achieved on slow-to-compute integrals (e.g. multidimensional ones).
The point of this test would be to see how much speedup more cores can bring. Thus you would need to adjust the number of threads Mathematica uses for a computation and see how many times faster operations complete on 200 cores than 10 cores (i.e. how much less than 20 times).
The following system options are interesting:
In[130]:= SystemOptions["ParallelOptions"]
Out[130]= {"ParallelOptions" -> {"AbortPause" -> 2.,
"BusyWait" -> 0.01, "MachineFunctionParallelThreshold0" -> 32768,
"MachineFunctionParallelThreshold1" -> 32768,
"MachineFunctionParallelThreshold2" -> 16384,
"MachineFunctionParallelThreshold3" -> 16384,
"MathLinkTimeout" -> 15., "MKLThreadNumber" -> 4,
"ParallelThreadNumber" -> 4, "RecoveryMode" -> "Retry",
"RelaunchFailedKernels" -> True}}
I believe that both "MKLThreadNumber" and "ParallelThreadNumber" are relevant. See SetSystemOptions on how to change them. I am not sure about their exact effect, but if you decide to do the test we can figure it out together.
Example benchmark:
We'll use this function to set the number of cores:
setThreadCount[
threads_] := (SetSystemOptions[
"ParallelOptions" -> "MKLThreadNumber" -> threads];
SetSystemOptions[
"ParallelOptions" -> "ParallelThreadNumber" -> threads];)
Eigenvalues for a dense symmetric matrix.
In[151]:= a = RandomReal[1, {5000, 5000}];
a = a + Transpose[a];
I have a laptop with a 4-core hyperthreading CPU. It appears as an 8-core one to the OS, but it only has 4 physical cores.
In[163]:= Table[
setThreadCount[t];
{t, First@AbsoluteTiming[Eigenvalues[a, 20];]},
{t, 1, 8}
]
Out[163]= {{1, 12.2941}, {2, 8.18082}, {3, 6.32726}, {4, 6.22842}, {5,
6.42646}, {6, 6.43579}, {7, 6.53777}, {8, 6.60184}}
Benchmarking is hard. Part of what we see here is due to the fact that as the CPU heats up it automatically throttles its clock rate. During the last few calculations it got hot so it has to slow down. If I repeat the 4-core test after waiting for it to cool down, I see better performance:
In[164]:= setThreadCount[4]
In[165]:= First@AbsoluteTiming[Eigenvalues[a, 20];]
Out[165]= 5.7838
AFAIK this won't be a problem with servers like the one you are testing.
But do pay attention to using only AbsoluteTiming and not Timing to measure parallel calculations. Timing measures the time used by each core separately then adds them up ...
BenchmarkReport[]and make the rest of us feel insignificant :-) $\endgroup$