Hot answers tagged parallel
35
Use these 3 components: compile, C, parallel computing.
Also to speed up coloring instead of ArrayPlot use
Graphics[Raster[Rescale[...], ColorFunction -> "TemperatureMap"]]
In such cases Compile is essential. Compile to C with parallelization will speed it up even more, but you need to have a C compiler installed. Note difference for usage of C and ...
35
Natively multi-threaded functions
A lot of functions are internally multi-threaded (image processing, numerical functions, etc.). For instance:
In[1]:= a = Image[RandomInteger[{0, 255}, {10000, 10000}], "Byte"];
In[2]:= SystemOptions["ParallelOptions"]
Out[2]= {"ParallelOptions" -> {"AbortPause" -> 2., "BusyWait" -> 0.01,
"MathLinkTimeout" ...
26
I second @Verbeia's suggestion: compute the function on a mesh of points and use ListContourPlot. The disadvantage is that ListContourPlot has no adaptive sampling, so it'd be preferable if we could do our own adaptive sampling somehow. Adaptive sampling can give you a much better result while needing to compute the function in far less points---and the ...
25
You could do something like this
mean = Reap[Sow @@@ Flatten[cogAK6, 1];, _, {Mean[#2], #} &][[2]];
This will be a lot faster than your approach because by using Sow and Reap this code only iterates through the list of data once. In your code, you reiterate through all elements of the data list for every value of gazeNo (so 3000 times instead of only ...
20
This might be an excellent candidate for ParallelTable;
MakeFractal[f_, nx_, ny_, {cx_, cy_}, {rx_, ry_}] := Module[{pts},
DistributeDefinitions[nx, ny, cx, cy, rx, ry, f];
pts = ParallelTable[f[x + I y],
{x, cx - rx, cx + rx, (2 rx)/nx},
{y, cy - ry, cy + ry, (2 ry)/ny}];
ArrayPlot[Reverse@pts, ColorFunction -> "TemperatureMap"]
]
...
18
There are two performance problems here. The first is relatively minor: MultinormalDistribution[μ, Σ] is evaluated in each slave kernel, returned to the master kernel, and sent back to the slave kernels as part of the RandomVariate call. In your example, this is a packed array of about 80KB in size: not large, yet not small either, and this behaviour may ...
17
This answer will be about efficiency, no ambitions for the beauty contest participation here. Since you mentioned speed, and only need mean values, here is a method that will be an order of magnitude faster and order of magnitude more memory-efficient than the (fine) solutions offered so far.
Code
First, we make a compiled function generator:
...
15
One way is to set a shared variable that would be assigned to an iterator variable, and monitor that:
SetSharedVariable[j]
Monitor[
ParallelTable[j = n;Length[FactorInteger[2^n - 1]], {n, 50, 300}],
j
]
This may make sense if the computation for each i is rather intensive, so that the overhead of communication with the main kernel is negligible. ...
13
Regarding using Sow instead of AppendTo, you may find this trick useful:
Last[Last[Reap[CheckAbort[Do[Pause[0.1]; Sow[x], {x, 30}], ignored]]]]
(Try running this and aborting it partway through. It runs for 3 seconds due to the Pause[0.1] commands.)
Do is used instead of Table, and the results are returned with Sow. The CheckAbort catches when you abort ...
13
Heike already got Sow/Reap so here is GatherBy:
<< cogAK6.m;
j = Join @@ cogAK6;
j[[All, 2]] = Round@j[[All, 2]];
{#[[1, 2]], Mean[#[[All, 1]]]} & /@ GatherBy[j, Last] // Sort
Both methods are fast because you do not repeatedly rescan the data for each gazeNo.
By the way I added Round just to be sure that values were properly gathered in case ...
12
In this case, you would want to favor Method->FinestGrained, not Method->CoarsestGrained.
FinestGrained is more useful when the workload has items that may take a very long time to process, or which may have wildly varying computation times.
CoarsestGrained is useful when the workload consists of many fast, easy to compute items. You don't want to ...
12
Use the function CriticalSection[{lck}, expr], for example:
CriticalSection[{logfileLock}, PutAppend[result, $runningLogFile]]
This way the result, if it's large, stays on its kernel. The other approach for synchronization is as ssch suggests, do things like this in the master kernel with shared functions, but then you need to ship the result from the ...
11
One idea, which was originally appealing to me, was to simply split the plot domain in four quarter size pieces and have these pieces calculated in parallel using ParallelTable and then combined using Show. It appears though that the overhead of getting the graphics data back to Show is pretty large, so this only yields some extra speed if the function is ...
11
While I cannot answer the general question, I would like to point out that Parallelize does more than replace certain functions with their parallel versions.
Consider this:
Parallelize[{f[$KernelID], f[$KernelID], f[$KernelID], f[$KernelID]}]
(* ==> {f[2], f[2], f[1], f[1]} *)
Parallelize[f[$KernelID] + f[$KernelID] + f[$KernelID] + f[$KernelID]]
(* ...
11
This message is issued by Function itself. To see this, try
Function[Module[{slot = 1}, Slot[slot]]]
If Function has named formal parameters, the message goes away:
Function[x, Module[{slot = 1}, Slot[slot]]]
So to fix this first we need to find out where is the argument passed to ParallelEvaluate wrapped by Function. Fortunately the parallel tools ...
10
The problem with SetSharedFunction is that it forces f to be evaluated on the main kernel: this means that if you simply do
SetSharedFunction[f]
then you will lose parallelization (a timing of ParallelTable[f[x], {x, 3}] will give about 9 seconds).
This property of SetSharedFunction is not clear from the documentation in my opinion. I learned about it ...
10
I will only address a subquestion of yours, namely, "Does ParallelTable send a copy of the memory content to each parallel kernel?". The answer to that is a clear no: using ParallelTable does not mean that each kernel gets a copy of everything, but only of a subset of things (see the docs for DistributeDefinitions to see how the set of symbols to be ...
10
First, I notice that you are using Real numbers such as 1. and 2. for Part indexes. While this works it would be better to use Integer indexes, 1 and 2.
Your use of PregaoMC and then Table, etc., is highly inefficient. Part and Span will be better. Observe:
Table[PregaoMC[x], {x, 2, n}] === CompleteMatrix[[2 ;; n, 1]]
True
n = 359835;
...
9
Note: While this whole post is just playing, the idea for the solution comes from an answer to a very real problem I had, so it wasn't just a futile exercise. The key is that (in v8) a shared function (set with SetSharedFunction) is always evaluated on the main kernel. Thanks to Andrew Moylan for pointing this out!
Here's my incredibly wasteful solution ...
9
Instead of driving a remote kernel directly via MathLink, you should consider implementing this task with webMathematica:
webMathematica 3 added support for queueing long running computations, which can also use compute kernels.
you can connect to webMathematica's web frontend with a web browser to inspect the state of the running computation at any time.
...
9
Why the unexpected result?
You quoted the documentation:
Parallelize[Table[expr,iter, …]] (which is equivalent to
ParallelTable[expr,iter,…]) will give the same results as Table,
except for side effects during the computation.
The explanation why the parallelized version returns an incorrect result is that your code does have side effects: you are ...
9
This limitation exists, and as far as I can see we have to live with it for now. It does indeed make it impractical to use Parallel operations inside functions---the functions won't work in parallel kernels.
I recommend you parallelize at the top level unless you have a specific reason to do it at lower levels. The longer the a "unit calculation" that is ...
9
As Daniel points out, you will always get a good speed-up by compiling. I want to take a different approach and consider how you can get a faster execution by writing your code in a more "functional" style.
My machine is slower than yours and ran your code in 17.2 seconds, according to AbsoluteTiming.
The first thing I noticed is that you are creating and ...
9
You appear to have access to 7 Mathematica licenses each of which allow running 4 sub Math kernels. This should allow you to address
4 + 28 = 32 kernels
Also of interest, each Mathematica network license comes with a home use license. You could allocate the home use licenses to additional machines (I think even virtual machines) and get another 4 ...
8
Here is an implementation that uses the scheduled tasks in version 8:
sleepSort[l_, t_:0.1] :=
Scan[RunScheduledTask[Print[#];RemoveScheduledTask[$ScheduledTask], {t #}]&, l]
In a notebook environment, output will appear in the messages window. The time interval is specified as the optional second argument, defaulting to 1/10th of a second (I'm ...
8
Many plots can be speeded up by pre-generating the data set you want and then plotting the resulting list. In any case, it's not coincidence that Table and Plot have similar syntax, only that Plot does additional things like finding out the range to be displayed, the interpolation strength, and so forth. If you're already sure what kind of picture you want ...
8
Here is my attempt to do this with a single kernel. Output is printed directly to the messages channel, and you cannot suppress the output of f /@ list because if the Dynamic expressions never display they never run.
list = RandomInteger[10, 10]
f = Module[{x},
(x := If[start + # < AbsoluteTime[], Print[#]; x =.]);
Dynamic[Refresh[x, ...
8
The best you can do it is to speed up your function. Your calc is using a replace, but it's better if you use With:
calc[a_, b_, c_, d_, e_, f_] :=
With[{Q = (8000000 \[Pi] Sin[
1/2 ArcCos[
1/2 Sqrt[Cos[2 c] + Cos[2 d]] Sqrt[Cos[2 e] + Cos[2 f]] +
Sin[c] Sin[e] +
Sin[d] Sin[f]]])}, (1.97531*10^15 (3. Q Cos[(3 ...
8
Here is another compiled effort (this will be ugly):
ClearAll[f]
f = Compile[
{
{vals, _Real, 1},
{ind, _Integer, 1}
},
Module[
{
s = ConstantArray[0., 4000],
n = ConstantArray[0, 4000]
},
Do[
s[[ind[[i]]]] += vals[[i]];
n[[ind[[i]]]]++,
{i, 1, Length@ind}
];
Transpose[{Drop[s, 999]/N@Drop[n, ...
8
Although I'd generally suggest using ParallelTable instead, I can imagine a scenario where you would like to use ParallelDo with a function that is really lengthy and could potentially hang or crash. Then you would want to save the results of each successful parallel computation to avoid losing it in case of a crash.
But to do that, it's safest to create a ...
Only top voted, non community-wiki answers of a minimum length are eligible
